- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The submission proposes a unified framework for embodied robot interaction. The input of the system is reference motion and text or music. The output would be a compatible follower motion, and transfers it to actual humanoid robots. The core component is SRG, which represents inter-agent geometric relationships and contact information across different stages. Experiments are conducted on two datasets, InterHuman and CoDance. They also present results on real-world robots.
Critical strengths
The whole framework is engineered together to a very practical (at least in its form) system, covering motion generation, embodiment-aware retargeting, and policy control. it is hard to say any part is completely brand-new, but the whole framework appears to be a useful system-level contribution.
Critical weaknesses
The experiment results need further justification.
First, the execution errors reported in the table are too large that I doubt on the effectiveness of interaction preservation. As shown in Table 4, the executed PRPE / DRGE reach 0.701 m / 0.909 m on InterHuman and 1.055 m / 1.166 m on CoDance. For an embodied humanoid I think an average errors about 1 meter are too large to claim the motion is successfully executed. Furthermore, Table 3 reports extremely large rotation tracking errors like 85.22° and 104.51°. I am not sure if there are any errors in calculation, or I have the wrong interpretation of this metrics.
Considering the large error, the improvements over existing baselines are marginal. For example, the executed DRGE on InterHuman only decreases from 0.941 m (GMR) to 0.909 m, while on CoDance, the executed PRPE decreases from 1.078 m to 1.055 m and DRGE from 1.191 m to 1.166 m. Given these large absolute errors and relatively minor improvements, I am not convinced that the proposed framework can reliably preserve the intended interactions during execution or have significant superiority over existing baselines. The authors should provide further clarifications.
The provided anonymous link provide video results, which is good, but I don't see comparisons with baseline methods. Also, some generation results appears significantly unnatural, such as the six "dance" examples in "part 1: Human motion generation".
Second, I feel that the contribution of this work are not clearly isolated. From my understanding, the central contribution of this paper is SRG, yet some of the existing papers already leveraged similar designs, like [1,2]. Also, in Tables 2 and 5, the model without SRG already substantially outperforms UMF in FID, and even receives better FID than the full model. I wonder what is the exact difference between the proposed model w/o SRG and UMF? This part needs further clarification.
Also, the main retargeting and execution experiments use 15 InterHuman pairs and 8 CoDance pairs, while the ablation experiments uses a different set of 17 InterHuman clips according to Table 10. The full model achieves much better results in the ablation than in the main experiments: PRPE decreases from 0.701 m to 0.1705 m, DRGE from 0.909 m to 0.2466 m. Could you please clarify why choosing different evaluation sets for main experiment and ablation study and why there are substantial performance gap?
[1] Rhythm: Learning interactive whole-body control for dual humanoids [2] InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs
Questions for authors
Please clarify the issues raised in the "Critical Weaknesses" part.
4 comments
Response to Reviewer K8ux
About video results. Thank you for your positive feedback regarding our video results. We have added several new sets of video results for comparison against the baseline. Regarding the CoDance results, we would like to provide some clarification: two-person dance sequences represent a particularly challenging area in this field, characterized by limited data, data sparsity, and high motion complexity. These factors make the transition from generation to execution quite difficult. We readily acknowledge that our two-person dance results appear unnatural; this work serves merely as an extension of our research and establishes a baseline for future studies. We do not claim the reconstruction of dance sequences as a core contribution of our paper.
Related Research. We have also taken note of relevant research from our peers. In addition to InterAgent [1]—which focuses on establishing interaction constraints during the generation phase—and Rhythm [2]—which addresses the maintenance of interaction relationships during the policy control phase—there is also the work Beyond Mimicry [3], which focuses on preserving two-person relationships during the redirection phase. Rhythm's official project page lists its code as "Coming Soon," and we have not located an official public implementation of Beyond Mimicry. Therefore, we cannot include them in the comparison. These recent studies are all discussed in the "Related Work" section of our paper. We do not claim to be the first to model or constrain interaction relations. Our contribution lies in using a shared relational graph representation to provide consistent relational guidance across motion generation, retargeting, and policy-based control through stage-specific operators.
Clarification regarding experimental data. For InterHuman and CoDance, we constructed two role-ordered examples from each motion pair by exchanging the leader and follower assignments. We will document this role-based data augmentation procedure in detail in the revised manuscript. This preprocessing leads to differences in evaluation scores relative to those reported in these original reference papers. Therefore, it is not appropriate to directly compare data from other papers with the experimental results presented in this paper. We apologize for any misunderstanding caused by missing description of data preprocessing, and we will include further clarification in the paper.
Explanation of the differences between PRPE and DRGE. As clarified in the appendix, Tables 4 and 7 use different motion sets and robot pairings. The main comparison uses G1–G1 pairs, whereas the subsequent ablation uses G1–T1 pairs with a correspondingly updated reinforcement-learning configuration. Different agents and different action sets are the reasons for significant performance differences. This change stems from hardware modifications in the laboratory. Since training multiple sets of new control strategies is extremely time-consuming, we did not rerun the main comparison under this configuration. However, we believe this issue should not affect the assessment of our main contribution.
[1]InterAgent: Physics-based multi-agent command execution via diffusion on interaction graphs.
[2]Rhythm: Learning interactive whole-body control for dual humanoids.
[3]Beyond mimicry: Learning whole-body human-humanoid interaction from human-human demonstrations.
We hope our explanation has cleared up your doubts and misunderstandings. We hope our Instructions allow our contribution to be assessed on the intended basis. We thank the reviewer for your thoughtful feedback.
Dear Reviewer,
Thank you for sharing your concerns. We recognize that our original presentation left room for misunderstanding and have provided detailed explanations in our response, with corresponding clarifications in the manuscript. The revised paper has been resubmitted, and our visualization website has also been updated. We believe the issues raised concern the clarity of our presentation rather than the novelty or substance of our contributions. We would be grateful if you could revisit our response and the revised manuscript and take these clarifications into account when assessing our main contributions.
Dear Reviewers,
We are now in the final stage of the rebuttal period. Thank you for your valuable review of our paper. We have submitted our rebuttal, addressing the concerns raised. Furthermore, we have incorporated additional experiments and clarifications into the paper, following the suggestions provided by you and the other reviewers. We believe that the problems raised stemmed from a misunderstanding of the experimental setup, which should not affect the assessment of our paper's main contribution.
We would be very grateful if you could take some time to review our response and re-evaluate the contributioin of our work. Thank you very much for your time and valuable contributions!
Response to Reviewer K8ux
Thank you very much for your careful review and for acknowledging that the whole framework appears to be a useful system-level contribution. Regarding your questions and some misunderstandings, we will provide clarification point by point.
Misconceptions regarding excessive execution errors. Execution errors are accumulated errors of 24 bodies. For example, if the table shows a global position error of 0.84 m and a rotation error of 85 degrees, this translates to an average error of 0.035 m and 3.54 degrees per joint. These are very small, reasonable margins of error that are entirely acceptable in a semantic context. We will include the method of calculation in the paper to avoid any unnecessary misunderstanding:
\frac{1}{TA} \sum_{t=1}^{T}\sum_{a=1}^{A}\sum_{b=1}^{B} ||x_{a,b,t}^{e}-x_{a,b,t}^{r}||_2.$$ $$E_l = \frac{1}{TA} \sum_{t=1}^{T}\sum_{a=1}^{A}\sum_{b=1}^{B} ||(x_{a,b,t}^{e}-x_{a,o,t}^{e})-(x_{a,b,t}^{r}-x_{a,o,t}^{r})||_2.$$ $$E_R = \frac{180}{\pi TA} \sum_{t=1}^{T}\sum_{a=1}^{A}\sum_{b=1}^{B} \arccos( \operatorname{clip}( \frac{\operatorname{tr}((R_{a,b,t}^{r})^T R_{a,b,t}^{e})-1}{2}, -1,1)).$$ Here, $E_g$, $E_l$, and $E_R$ denote global position error, local position error, and rotation error, respectively. T is the number of evaluated frames, A is the number of evaluated agents, and B is the number of tracked bodies per agent. The indices a, b, and t identify the agent, body, and frame, while o identifies the root body. The variables x and R represent world-frame position vectors and rotation matrices, respectively; superscripts e and r denote executed motion and the retargeted reference. The operator tr denotes the matrix trace. **About improvements over existing baselines are marginal.** The DRGE and PRPE metrics you mentioned are evaluation parameters defined in our paper, calculated using Equation (7). Since these metrics are averaged over each semantic edge, they are highly sensitive to interactions. Interaction edges include relationships between agents that span across their respective bodies (edges that do not involve physical contact), which results in a larger base count for these parameters. Furthermore, we believe that the contributions of DyadLink are not limited to these two evaluation metrics. Its effectiveness is demonstrated through the various quantitative results presented in the paper, as well as the visualization results results provided on the anonymous website referenced in the paper. In addition, we provide a set of comparative results covering three distinct stages to demonstrate the effectiveness of DyadLink throughout the entire process, as shown in the table below; the corresponding visualizations can be viewed on an anonymous website. **Whole pipeline comparison on InterHuman.** Each downstream stage takes the output of the preceding stage as its input. Bold indicates the best mean for each metric. | Method | Generation: FID ↓ | Generation: Contact F1 ↑ | Retargeting: PRPE (m) ↓ | Retargeting: DRGE (m) ↓ | Retargeting: Global pos. (m) ↓ | Retargeting: Contact F1 ↑ | Control: Executed PRPE (m) ↓ | Control: Executed DRGE (m) ↓ | Control: Contact Recall ↑ | Control: Contact F1 ↑ | | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Others | 4.432 ± 0.032 | 0.141 ± 0.012 | 0.65 ± 0.02 | 0.82 ± 0.08 | 0.9 ± 0.08 | 0.322 ± 0.025 | 1.02 ± 0.12 | 1.14 ± 0.11 | 0.362 ± 0.258 | 0.404 ± 0.204 | | **Ours (DyadLink)** | **3.759** ± 0.043 | **0.194** ± 0.004 | **0.15** ± 0.04 | **0.18** ± 0.06 | **0.86** ± 0.12 | **0.415** ± 0.012 | **0.71** ± 0.05 | **0.89** ± 0.06 | **0.698** ± 0.205 | **0.702** ± 0.103 | **Others** combines InterAgent[1] for generation, OmniRetarget[2] for retargeting, and BeyondMimic[3] for control. The retarget and control stage employs six motions sampled from a generative model. Some results in the new table differ from those in the original manuscript because the evaluations follow different input protocols. In the whole comparison, each downstream stage receives the output of the preceding stage. By contrast, the stage-wise evaluations use controlled reference inputs, as described in the manuscript, to isolate the contribution of SRG at each stage. [1]InterAgent: Physics-based multi-agent command execution via diffusion on interaction graphs. [2]OmniRetarget: Interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction. [3]BeyondMimic: From motion tracking to versatile humanoid control via guided diffusion.