acceptodds
Under review as a conference paper at ICLR 2027

OmniInteract: Towards Unified Whole-Body Interaction Control for Dual Humanoids

Abstract

Physical interaction is a key capability for humanoid collaboration, requiring coordinated whole-body motion and precise contact. However, learning diverse dual-humanoid interactions faces two major challenges: (1) human-robot morphological differences disrupt contact interfaces and spatial relationships during retargeting, and (2) jointly learning motor skills and partner coordination makes it difficult to retain control fidelity as interaction diversity grows. In this paper, we propose to address these challenges through surface-aware modeling and scalable residual policy learning. First, we introduce , which leverages dense human-robot surface correspondences to jointly retarget both humanoids, preserving explicit interaction contact interfaces and their near-field geometric context. Second, we propose a for partner-conditioned residual generation. It factorizes dual-humanoid control into a reusable motor foundation and interaction-specific adaptations: partner-aware experts learn corrections over a frozen whole-body motion prior, while conditional flow matching unifies them in a shared residual action space. This enables scalable skill consolidation while retaining pretrained motor capabilities. Extensive experiments demonstrate that OmniInteract improves retargeting fidelity and dual-humanoid coordination while better retaining interaction performance as skill diversity grows. Real-world deployment further demonstrates successful contact establishment, stable whole-body coordination, and recovery from external perturbations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid, Learning-Based Control, Motion Retargeting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HzxsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniInteract, a framework for transferring human-human interaction demonstrations to collaborative control of dual humanoid robots. The method consists of two stages. First, Surface Contact-Aware Retargeting (SCAR) jointly optimizes the poses of two humanoid robots while preserving motion fidelity and interaction constraints. Second, a partner-aware residual control framework trains task-specific policies using MAPPO on top of a frozen motion tracker, then distills them into a shared generative controller using DAgger and Conditional Flow Matching. Experiments on InterHuman and Inter-X datasets, unseen motions, and real Unitree G1 robots demonstrate improved retargeting performance and successful collaborative interactions.

Critical strengths
  • The implementation details are well documented, and the supplementary material and real-world experiments are valuable additions.
  • The paper is well written and easy to follow, with a clear structure and well-explained methodology.
Critical weaknesses
  • Insufficient comparison with directly related dual-humanoid interaction methods. In particular, It Takes Two[1] presents contact-aware retargeting and interaction-driven control for two humanoids. More importantly, Rhythm[2] proposes a complete dual-humanoid interaction framework that includes jointly constrained interaction retargeting, graph-based interaction rewards, MAPPO, and real-world execution on Unitree G1 robots.
Questions for authors
  • (Main Concern) How does SCAR compare quantitatively with Rhythm[2]'s IAMR and a coupled version of OmniRetarget[3] under identical datasets and metrics?
  • Could the authors provide a capacity-matched deterministic Transformer or MLP baseline to isolate the contribution of Flow Matching?
  • How does the number of flow-matching Euler steps during inference affect performance?
  • The motion preprocessing removes finger articulation, but some tasks involve hand contact or grasping. How does this affect interaction fidelity, and can the method extend to humanoids with dexterous hands?

[1] Liu et al., It Takes Two: Learning Interactive Whole-Body Control Between Humanoid Robots, 2025.

[2] H. Chen et al., Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids, Robotics: Science and Systems (RSS), 2026.

[3] L. Yang et al., OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction, IEEE International Conference on Robotics and Automation (ICRA), 2026.

7 comments

Response to Reviewer Hzxs (Part 1 / 2)

We really appreciate the reviewer for the detailed and constructive feedback. We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:


Q1:Comparison with directly related dual-humanoid interaction methods.

A1: Thank you for highlighting these related dual-humanoid interaction methods. To the best of our knowledge, neither It Takes Two [1] nor Rhythm [2] has publicly released its official implementation to date. Nevertheless, we have fully re-implemented their retargeting and control pipelines following the original papers. We further extended OmniRetarget [3] to a coupled dual-agent setting to provide a more comprehensive comparison under the same datasets and evaluation metrics in R3-Table1.

R3-Table1: Quantitative comparison with explicitly interaction-aware dual-humanoid retargeting baselines.

DatasetMethodMPD (cm) ↓CIE (%) ↓NGE (%) ↓C-F1 (%) ↑DSR (%) ↑
InterHumanIAMR (Rhythm)0.725.956.5464.0370.4
Harmanoid-Retarget3.745.697.1769.3564.1
Dual OmniRetarget1.287.957.1854.7561.8
SCAR (Ours)0.934.966.4965.1875.4
Inter-XIAMR (Rhythm)0.675.334.8660.6266.2
Harmanoid-Retarget4.855.175.0361.7163.6
Dual OmniRetarget1.687.066.2653.8160.8
SCAR (Ours)1.203.974.9964.0872.1

R3-Table2: Quantitative comparison with dual-humanoid controllers and variants on benchmark and unseen motions.

MethodInterHuman† + Inter-X†Unseen Motions
Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑
Harmanoid62.318.8764.2773.5611.8759.45
Rhythm58.347.2371.1966.899.8965.91
OmniInteract (Ours)41.676.4284.3261.678.9272.89

The added quantitative results and qualitative visualizations are highlighted in Table 1 and Figure 3 of the revised manuscript. SCAR achieves the lowest CIE and highest DSR on both datasets, improving DSR over the strongest added baseline by 5.0% and 5.9% on InterHuman and Inter-X, respectively. The complementary controller comparison shows that OmniInteract exceeds Rhythm’s success rate by 13.13% and 6.98% on benchmark and unseen motions, respectively.

These results further support the effectiveness of our design: SCAR explicitly models fine-grained surface contacts and their surrounding geometry, while OmniInteract scalably consolidates partner-aware residual skills over a frozen motion prior for unified dual-humanoid control.


Q2: Capacity-matched experiments (Flow Matching vs deterministic baselines).

Thank you for this question about capacity-matched comparison. We have added deterministic MLP baselines with 30M, 50M, and 80M parameters, as shown in R3-Table3. All variants use the same training data and evaluation protocol.

R3-Table3: Capacity-matched comparison of OmniInteract with deterministic MLP baselines on InterHuman† + Inter-X†.

Parameters (M)MethodEmpjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑
30Deterministic MLP48.017.3880.19
30OmniInteract (Flow Matching)48.377.6679.48
50Deterministic MLP49.367.9378.52
50OmniInteract (Flow Matching)43.986.7182.64
80Deterministic MLP46.237.2980.68
80OmniInteract (Flow Matching)41.676.4284.32

The MLP performs slightly better at 30M. However, at 50M and 80M, Flow Matching achieves better results across all three metrics, improving success rates by 4.12% and 3.64%, respectively. As capacity increases from 30M to 80M, Flow Matching’s success rate improves by 4.84%, compared with 0.49% for the MLP.

These capacity-controlled results show that OmniInteract’s gains cannot be explained by parameter count alone and support the effectiveness of Flow Matching in utilizing larger model capacity for unified interaction control.

Response to Reviewer Hzxs (Part 2 / 2)


Q3: Effect of Euler step count on control performance and inference latency.

A3: Thank you for raising this inference-time trade-off. We evaluate 1, 3, 5, 8, and 10 Euler steps in R3-Table4 using the same checkpoint and evaluation protocol, with batch size 1. Increasing the Euler step count from 1 to 5 improves all three control metrics, raising success from 70.56% to 84.32%. Further increasing to 10 steps yields diminishing returns: success improves by just 0.69% and MPJPE decreases by 1.02 mm, while inference latency nearly doubles—from 9.8 to 18.6 ms on the Orin NX and 4.0 to 7.5 ms on the RTX A6000.

R3-Table4: Effect of Euler step count on OmniInteract’s control performance and inference latency.

Euler stepsInterHuman† + Inter-X†Inference policy latency ↓
Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑Unitree G1 Jetson Orin NX (ms)NVIDIA RTX A6000 (ms)
156.288.7970.56≈2.2≈0.9
346.787.5479.51≈6.2≈2.5
5 (default)41.676.4284.32≈9.8≈4.0
840.896.2284.78≈14.2≈5.7
1040.656.1585.01≈18.6≈7.5

We therefore retain 5 Euler steps as the default, providing a favorable balance between control accuracy and inference latency.


Q4: Finger simplification and extension to dexterous hands.

A4: Thank you for raising this concern. We conducted a paired ablation on 20 complete Inter-X sequences (2,987 frames), varying source finger articulation while keeping the fixed-hand G1 target and SCAR optimization settings unchanged. As shown in R3-Table5, preserving source finger articulation reduces hand-contact vector error by 32.3% and improves hand-endpoint F1 by 11.28%. These results show that source finger geometry matters for contact preservation even with a fixed-hand target.

R3-Table5: Effect of source finger articulation on hand-contact fidelity with a fixed-hand G1 target.

Source finger inputHand-contact vector error (mm) ↓Hand-endpoint F1 (%) ↑
Neutralized70.7826.52
Original articulation47.9437.80

Both variants are evaluated against the original articulated source using a 20 mm contact threshold.

We also extended our method to G1 with Inspire dexterous hands by retaining source finger poses and binding palm and finger-surface samples to the corresponding robot links. The bodies and fingers of both humanoids are jointly optimized using the same contact-interface and near-field objectives, subject to joint-limit, joint-coupling, velocity, and collision constraints. We have added the visualization of our method with dexterous hands in the revised manuscript (please see Figure 12).


We are happy to see from your comments that you have an overall positive view of our work, and we respectfully ask if you'd be comfortable to adjust your ratings, based on all the new experiments and justification to your questions. Thank you once again for your time and effort. We look forward to incorporating these insightful discussions into the final camera-ready version.

[1] Liu et al., It Takes Two: Learning Interactive Whole-Body Control Between Humanoid Robots, 2025.

[2] H. Chen et al., Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids, Robotics: Science and Systems (RSS), 2026.

[3] L. Yang et al., OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction, IEEE International Conference on Robotics and Automation (ICRA), 2026.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Reviewer_9QnFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OmniInteract, a framework for learning dual-humanoid interaction skills. It combines Surface Contact-Aware Retargeting with a scalable flow-based interaction policy to preserve interaction geometry and learn coordinated control.

Critical strengths
  • The paper is clearly written and well-organized.
  • The experiments demonstrate the practical value of the overall system and provide evidence for the effectiveness of the proposed components.
Critical weaknesses
  • The retargeting comparisons do not separate the benefit of explicitly modeling interactions from that of incorporating surface geometry. All baselines, as implemented in this paper, lack explicit inter-humanoid interaction modeling. The comparisons should at least include prior methods such as Harmanoid and Rhythm.
  • The novelty of the proposed flow-based interaction policy appears limited. MAPPO, distillation into flow-based policies, and learning in a residual action space are all established techniques. The experiment in Sec 4.3 seems less informative. The paper does not clearly establish a methodological contribution beyond adapting these techniques to dual-humanoid control.

Minor comment

  • The voice-over audio in the demo video sounds choppy and could be improved.
Questions for authors
  • How is E_inter-geo computed from CIE and NGE? This definition is needed to interpret the results in Table 2 and assess whether OmniInteract adequately preserves interactions in the simulation tests.
6 comments

Response to Reviewer 9QnF (Part 1 / 3)

We sincerely appreciate your careful assessment and valuable suggestions. We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:


Q1: Interaction-aware baseline comparisons and the contribution of surface-level modeling.

A1: Thank you for highlighting these related dual-humanoid interaction methods. To the best of our knowledge, neither It Takes Two [1] nor Rhythm [2] has publicly released its official implementation to date. Nevertheless, we have fully re-implemented their retargeting and control pipelines following the original papers. We further extended OmniRetarget [3] to a coupled dual-agent setting under the same datasets and evaluation metrics in R2-Table1.

The added Harmanoid-Retarget, IAMR (Rhythm), and Dual OmniRetarget baselines all explicitly model inter-humanoid interactions. In particular, IAMR and Dual OmniRetarget provide sparse-body interaction comparators for SCAR’s explicit surface-contact and near-field context formulation.

R2-Table1: Quantitative comparison with explicitly interaction-aware dual-humanoid retargeting baselines.

DatasetMethodMPD (cm) ↓CIE (%) ↓NGE (%) ↓C-F1 (%) ↑DSR (%) ↑
InterHumanIAMR (Rhythm)0.725.956.5464.0370.4
Harmanoid-Retarget3.745.697.1769.3564.1
Dual OmniRetarget1.287.957.1854.7561.8
SCAR (Ours)0.934.966.4965.1875.4
Inter-XIAMR (Rhythm)0.675.334.8660.6266.2
Harmanoid-Retarget4.855.175.0361.7163.6
Dual OmniRetarget1.687.066.2653.8160.8
SCAR (Ours)1.203.974.9964.0872.1

As shown in R2-Table1, SCAR achieves the lowest CIE and highest DSR on both datasets. Relative to IAMR, CIE decreases from 5.95% to 4.96% on InterHuman and 5.33% to 3.97% on Inter-X, while DSR improves by 5.0% and 5.9 %, respectively. Thus, SCAR’s advantages persist against explicitly interaction-aware methods. These results support the additional value of SCAR’s combined surface-contact and near-field formulation over existing interaction-aware alternatives. The added comparisons and visualizations are highlighted in Table 1 and Figure 3 of the revised manuscript.


Response to Reviewer 9QnF (Part 2 / 3)


Q2: Architectural novelty and experimental validation of the interaction policy.

A2: Thanks for the good question about architectural contribution and the experimental evidence. We would like to emphasize our key insight here: OmniInteract extends a pretrained motion controller into a unified dual-humanoid interaction controller while retaining its motor foundation. The contribution lies in how the control problem is decomposed and learned.

Our architecture separates two learning problems. First, interaction learning is separated from relearning individual motion control. A frozen prior supplies the base actions, while partner-aware experts learn the adaptations required under coupled dynamics. Second, acquiring individual interaction skills is separated from constructing a policy that covers many motions. Task-specific RL develops specialized coordination skills, and a conditional flow policy consolidates them over the same frozen foundation. This avoids requiring one RL policy to discover all interaction behaviors simultaneously, while retaining a single unified interaction policy for deployment.

To make Sec. 4.3 more informative, we have added Harmanoid, Rhythm, and Frozen SONIC + Residual MAPPO to the control comparisons, with evaluations on both benchmark and unseen motions (R2-Table2). These additions examine two distinct alternatives:

  • From-scratch interaction learning. Harmanoid and Rhythm are also from-scratch RL baselines, incorporating specialized contact and geometric interaction objectives. OmniInteract outperforms all three across every reported metric. Against Rhythm, the strongest baseline in this group, success increases from 71.19% to 84.32% on benchmark motions and 65.91% to 72.89% on unseen motions.

  • A frozen motion prior with residual RL. Frozen SONIC + Residual MAPPO already contains the pretrained motor foundation and residual actions. OmniInteract further improves success from 74.23% to 84.32% and 66.55% to 72.89%, respectively, while reducing tracking and interaction-geometry errors. These gains support the value of the complete architecture beyond simply introducing a strong prior and residual learning.

R2-Table2: Quantitative comparison with dual-humanoid controllers and variants on benchmark and unseen motions.

MethodInterHuman† + Inter-X†Unseen Motions
Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑
Harmanoid62.318.8764.2773.5611.8759.45
Rhythm58.347.2371.1966.899.8965.91
Frozen SONIC + Residual MAPPO51.567.9974.2365.2610.2666.55
From-Scratch RL59.938.5468.4968.7110.8363.78
OmniInteract (Ours)41.676.4284.3261.678.9272.89

OmniInteract achieves the best results across all six evaluation columns. The expanded comparisons substantiate the advantages of preserving motor competence, learning interaction adaptations separately, and consolidating them into a unified controller, with improvements that extend to unseen motions. We have revised Sec. 4.3 (please see the revised manuscript) to clarify the effectiveness of our framework.


Q3: Voice-over quality in the demo video.

A3: Thank you for pointing this out. We have revised the voice-over to improve its fluency and clarity and included the updated demo video in the supplementary materials.

Response to Reviewer 9QnF (Part 3 / 3)


Q4: Definition and interpretation of Einter-geoE_{\text{inter-geo}} in Table 2.

A4: Thank you for pointing this out. In Table 2, Einter-geoE_{\text{inter-geo}} measures how well the executed robot motion preserves the interaction geometry of the retargeted reference, combining contact-interface and near-field errors with equal weights:

Einter-geo=12(CIEctrl+NGEctrl).E_{\text{inter-geo}} = \frac{1}{2}\left(\mathrm{CIE}_{\mathrm{ctrl}} + \mathrm{NGE}_{\mathrm{ctrl}}\right).

For corresponding surface points on the two robots, we first compute the error between their executed and reference relative vectors:

eij,t=∥(pi,tA−pj,tB)exec−(pi,tA−pj,tB)ref∥2.e_{ij,t} = \left\lVert (\mathbf{p}_{i,t}^{A} - \mathbf{p}_{j,t}^{B})^{\mathrm{exec}} - (\mathbf{p}_{i,t}^{A} - \mathbf{p}_{j,t}^{B})^{\mathrm{ref}} \right\rVert_2.

We then average this error separately over contact pairs and near-field pairs, and normalize both by the same body scale L=1.32,mL = 1.32\\,\mathrm{m}:

CIEctrl=100LeˉC,NGEctrl=100LeˉN.\mathrm{CIE}_{\mathrm{ctrl}} = \frac{100}{L}\bar{e}_{\mathcal{C}}, \qquad \mathrm{NGE}_{\mathrm{ctrl}} = \frac{100}{L}\bar{e}_{\mathcal{N}}.

The means include all eligible frame–pair samples within each rollout. Both sets are determined solely from the reference: contact pairs have reference distances within 2 cm, while near-field pairs have reference distances between 2 and 15 cm, excluding pairs whose endpoints both participate in contact. This ensures that contacts lost during execution remain evaluated.

The interaction-geometry metric in Table 2 complements individual motion-tracking accuracy, with lower values indicating more faithful preservation of the intended interaction geometry. Unlike the retargeting metrics in Table 1, which compare human and retargeted robot motions, its components measure discrepancies between reference and executed robot motions. Scores are computed for each rollout and averaged across evaluation trials.


We appreciate your recognition of our system’s practical value, and we respectfully ask if you'd be comfortable to adjust your ratings, based on all the new experiments and clarification to your questions. Thank you once again for your time and effort. We look forward to incorporating these insightful discussions into the final camera-ready version.

[1] Liu et al., It Takes Two: Learning Interactive Whole-Body Control Between Humanoid Robots, 2025.

[2] H. Chen et al., Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids, Robotics: Science and Systems (RSS), 2026.

[3] L. Yang et al., OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction, IEEE International Conference on Robotics and Automation (ICRA), 2026.

Reviewer_Poz1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper makes two contributions to improve the contact fidelity in dual-humanoid interactive motion retargeting, with their proposed framework called OmniInteract. It uses a novel retargeting algorithm, Surface Contact-Aware Retargeting (SCAR), that optimizes retargeted robot motions using a heuristic-based method that determines surface contacts between the two interactive human motions. It then trains an interactive expert for each motion using residual reinforcement learning on top of SONIC. Lastly, it unifies multiple of these learned RL experts into a unified flow-matching policy, which distilled the motion priors from these RL experts through DAgger. For experiments, the authors study 1. how well SCAR retargets among other baseline retargeting algorithms, 2. if SONIC improves the residual RL learning compared to learned from scratch, 3. if the flow policy scale with more RL experts 4. how OmniInteract performs and handles perturbations in the real world.

Critical strengths
  1. The authors performed extensive experiments that benchmarks current state-of-the-art whole-body retargeting algorithms with the proposed SCAR. The experiment results show that SCAR outperforms all other evaluated baselines across multiple dimensions, such as feasibility, interaction fidelity, and utility.

  2. The authors proposed a novel, unified method by distilling the RL motion experts into a single residual flow-matching policy that scales. The formulation is clear, demonstrating the method's generalization to unseen motions.

Critical weaknesses
  1. The real-world robustness experiment setup lacks a controlled setting, a standardized disturbance protocol, and detailed scoring metrics.

  2. Section 4.3 is confusing and needs more justification to isolate what makes residual learning from frozen motion priors effective. While from-scratch policies are included in the experiments, the policy-learning procedures differ, and the comparisons should be conducted more fairly under a fixed compute budget.

Questions for authors
  1. For seen motions, what are some tradeoffs for using index-based RL experts and why would you prefer the unified flow-matching policy over it?

  2. We are training/inference-time compute costs, inference latency of this method?

  3. Further clarification of section 4.3 and justification to isolate what makes residual learning from frozen motion priors effective.

  4. One strong assumption that this work builds on is that the SONIC tracker is robust to build residual policies on top of. Is OmniInteract generalizable when building on top of other motion tracker asides from SONIC? Does the robustness still hold and the performance gain exists in these other trackers too?

  5. The unified student policy is a flow-based policy. In terms of performance, how well would a diffusion-based policy perform under OmniInteract when scaling up number of distlled motions when using the same compute as flow-matching? Does the result hold when having different levels of compute budget?

4 comments

Response to Reviewer Poz1 (Part 1 / 3)

We thank the reviewer for the thoughtful and constructive feedback. We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication. Our point-by-point responses are provided below.


Q1: Experimental evaluation protocol for real-world robustness.

A1: Thank you for this question. We clarify that the manual-push experiment is intended as a qualitative demonstration of real-world interaction adaptation, consistent with the disturbance demonstrations in Rhythm [1] (Sec. IV-E, Fig. 6). In our arm-grabbing example, an external push disrupts the robots’ relative configuration, then the robots subsequently adjust their poses and establish the intended contact, illustrating online correction of interaction misalignment on physical hardware. For quantitative real-world evaluation, we follow the keyframe-based contact-establishment criterion used in Rhythm [1] (Table III), reporting success rates for High-Five, Hug, and Shoulder Hold. These success rates measure task-specific contact establishment, while the push demonstration illustrates the controller’s response to external interference.


Q2: Justification and fair evaluation of residual learning from frozen motion priors in Sec. 4.3.

A2: Thank you for highlighting the need for a fairer and more informative evaluation in Sec. 4.3. To address these concerns, we expanded the evaluation with directly related dual-humanoid controllers and a targeted residual-learning comparison, as shown in R1-Table1.

R1-Table1: Quantitative comparison with dual-humanoid controllers and variants on benchmark and unseen motions.

MethodInterHuman† + Inter-X†Unseen Motions
Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑
Harmanoid62.318.8764.2773.5611.8759.45
Rhythm58.347.2371.1966.899.8965.91
Frozen SONIC + Residual MAPPO51.567.9974.2365.2610.2666.55
From-Scratch RL59.938.5468.4968.7110.8363.78
OmniInteract (Ours)41.676.4284.3261.678.9272.89

Frozen SONIC + Residual MAPPO and From-Scratch RL use the same policy-learning procedure and the same fixed compute budget for interaction training. From-Scratch RL learns complete control actions, whereas Frozen SONIC + Residual MAPPO learns corrections over the retained motion controller. Under this matched setting, residual adaptation increases success from 68.49% to 74.23% on benchmark motions and 63.78% to 66.55% on unseen motions, while reducing both tracking and interaction-geometry errors. These results support the effectiveness of retaining pretrained motor competence when learning interaction coordination.

Rhythm [1] and Harmanoid [2] further strengthen the from-scratch comparisons through specialized contact and geometric interaction objectives. OmniInteract outperforms these methods across all reported metrics and reaches 84.32% / 72.89% success on benchmark/unseen motions. We have revised Sec. 4.3 (please see the revised manuscript) to clarify the effectiveness of frozen-prior residual learning through controlled, compute-matched comparisons.


Response to Reviewer Poz1 (Part 2 / 3)


Q3: Tradeoffs between indexed experts and a unified interaction policy.

A3: Thank you for this question. Our goal is to retain specialist-level execution quality while extending control beyond a fixed motion library. Indexed experts require selecting a motion-specific controller. A unified policy instead conditions directly on the interaction state and motion reference, allowing the same controller to handle both seen and previously unseen motions. R1-Table2 quantifies the accuracy cost of this consolidation.

R1-Table2: Comparison on 60 seen motions from InterHuman† + Inter-X†. Metrics are macro-averaged across motions.

MethodEmpjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑
Indexed RL experts40.126.0785.62
Specialist → Unified MLP46.237.2980.68
OmniInteract (Unified Flow Matching)41.676.4284.32

On seen motions, OmniInteract remains within 1.30% of expert success, with MPJPE increasing by 1.55 mm and inter-geometry error from 6.07 to 6.42. It also exceeds the unified MLP by 3.64% in success, showing that flow-based consolidation better preserves specialist performance. The same unified policy achieves 72.89% success on unseen motions without training dedicated experts for those motions. This demonstrated capability beyond the indexed library motivates our choice. For deployment restricted to known motions with available IDs, indexed experts remain a strong alternative.


Q4: Training cost and inference efficiency.

A4: Thank you for this question about computational efficiency. The training cost and inference efficiency of OmniInteract are summarized in R1-Table3.

R1-Table3: Training cost and inference efficiency of OmniInteract. Inference includes SONIC and the unified residual policy, using 5 Euler steps and batch size 1.

StageHardwareTraining cost (GPU-hours)Peak inference memory (GiB)End-to-end policy latency (ms) ↓
Training: experts + unified student2 × NVIDIA RTX A600040N/AN/A
InferenceNVIDIA RTX A6000N/A1.16≈4
InferenceUnitree G1 Jetson Orin NXN/A1.16≈9.8

The complete policy runs at approximately 9.8 ms on G1’s onboard hardware, below the 20 ms control period at 50 Hz, with a peak inference memory footprint of 1.16 GiB.


Q5: Does OmniInteract generalize beyond the SONIC tracker?

A5: Thank you for this good question. To examine this assumption, we evaluated OmniInteract with TWIST2 [3] as an alternative motion tracker. R1-Table4 compares TWIST2 with and without OmniInteract on benchmark and unseen motions.

R1-Table4: OmniInteract with TWIST2 as the base tracker.

MethodInterHuman† + Inter-X†Unseen Motions
Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑Empjpe↓E_{\mathrm{mpjpe}} \downarrowEinter-geo↓E_{\text{inter-geo}} \downarrowSucc. (%) ↑
TWIST286.5611.4441.36113.3514.7228.58
TWIST2 + OmniInteract68.548.9268.2479.1210.7352.15

OmniInteract improves TWIST2 across all reported metrics, increasing success by 26.88% on benchmark motions and 23.57% unseen motions, while reducing both tracking and interaction-geometry errors. These improvements persist despite TWIST2’s lower baseline performance and extend to unseen motions. The gains are therefore not specific to SONIC, supporting the applicability of our residual interaction-learning framework across different motion trackers.


Response to Reviewer Poz1 (Part 3 / 3)


Q6: Diffusion vs. Flow Matching across motion scales and compute budgets.

A6: Thank you for this question. We added a diffusion-student comparison across motion scales and sampling budgets, as shown in R1-Table5.

R1-Table5: Success rates (%) of Diffusion vs. Flow Matching across motion scales and sampling budgets (default: 5 steps).

Method10 motions30 motions60 motions60 motions (20 sampling steps)
DDPM (DDIM sampling)89.387.283.186.9
Flow Matching (Euler, Ours)90.188.684.386.8

Flow Matching leads by 0.8–1.4% across motion scales under the default settings. With 20 sampling steps, DDPM and Flow Matching achieve 86.9% and 86.8%, respectively. Thus, Flow Matching has a modest advantage at the default sampling budget, while diffusion becomes equally competitive with more inference computation.


We are pleased that you recognize our extensive experiments and methodological novelty, and we respectfully ask if you'd be comfortable to adjust your ratings, based on all the new experiments and justification to your questions. Thank you once again for your time and effort. We look forward to incorporating these insightful discussions into the final camera-ready version.

[1] H. Chen et al., Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids, Robotics: Science and Systems (RSS), 2026.

[2] Liu et al., It Takes Two: Learning Interactive Whole-Body Control Between Humanoid Robots, 2025.

[3] Ze et al., TWIST2: Scalable, Portable, and Holistic Humanoid Data Collection System, IEEE International Conference on Robotics and Automation (ICRA), 2026.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Comments on the paper

1 comment

General Response: New Experiments, Clarifications and Summary of Revisions

We sincerely thank Reviewers Poz1 (R1), 9QnF (R2) and Hzxs (R3) for their thoughtful and constructive feedback. We appreciate their recognition of our framework’s novelty (R1), our system’s practical value (R2), and the clarity of our formulation (R1,R2,R3).

We have carefully addressed each individual comment provided by the reviewers and believe we have successfully responded to most of their concerns. In our revised manuscript, we have incorporated the suggested experiments, additional discussions, and relevant updates to further strengthen our work. Below, we summarize the core contributions of our study, the updates to our experiments, and the in-depth discussions included in our revision.


Core Contributions of Our Work

  1. Surface Contact-Aware Retargeting (SCAR): We jointly optimize both humanoids using dense surface correspondences, explicitly preserving the contact interface and its surrounding spatial organization through complementary contact-interface and near-field objectives.

  2. Scalable Partner-Aware Residual Control: We separate interaction learning from relearning individual motor skills. Task-specific experts learn partner-aware corrections over a shared, frozen motion prior, and a unified flow policy consolidates these skills through DAgger. This architecture supports diverse interactions and generalization to unseen motions while retaining pretrained motor abilities.

  3. Extensive Evaluation and Discussion: We conduct comprehensive evaluations on InterHuman and Inter-X, including baseline comparisons, ablations, and analyses of scalability and generalization. Deployment on two physical Unitree G1 robots demonstrates coordinated whole-body interactions, successful contact establishment, and adaptation to external disturbances, providing practical evidence beyond simulation.


Updates of experimental results during Rebuttal

  • Main Text (Table 1, Figure 3 and Table 2): Added interaction-aware retargeting and control baselines (R2,R3) along with a comparison between residual learning (Residual MAPPO) and training from scratch under a fixed compute budget (R1).

  • Appendix D.3: Added experiments with TWIST2 as the base tracker to evaluate OmniInteract’s applicability beyond SONIC (R1).

  • Appendix D.4.: Added comparisons between diffusion and flow-matching students across motion scales and sampling budgets (R1).

  • Appendix D.5: Added an Euler-step ablation to examine the trade-off between control performance and inference latency (R3).


Updates of in-depth and addtional discussions during Rebuttal

  • Main Text (Sec. 4.3): Clarified Sec. 4.3 and strengthened the justification for the effectiveness of residual learning from frozen motion priors (R1,R2).

  • Appendix C.3: Clarified the definition of Einter-geoE_{\text{inter-geo}} and its interpretation in Table 2 (R2).

  • Supplementary Materials: Updated the demo video’s voice-over to improve fluency and clarity (R2).


We believe these additions and clarifications comprehensively address the reviewers' concerns and enhance the overall quality of our manuscript. All revisions are highlighted in blue-colored text for ease of reference. Our manuscript is updated on Nov 18, AOE time.

We look forward to the reviewers' favourable consideration and remain grateful for their valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.