- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This submission proposes CoLM, which takes text descriptions, object geometry, and the initial states of both the robot and the object as inputs to generate and execute manipulation tasks for a humanoid robot. It supports spatial control, ranging from target object poses to complete object trajectories. The method begins by retargeting six publicly available human-object interaction datasets for the Unitree G1 robot, recovering lost contact information through contact-guided refinement to construct a dataset of approximately 10,000 sequences. A diffusion model is then trained to jointly generate robot motion, object motion, and an "interaction field", a dense representation describing the relative geometric relationship between the robot and the object. Finally, a reinforcement learning-based tracking policy is employed to execute the generated actions. The interaction field serves both as a geometric consistency constraint during the generation phase and as a reward signal during the tracking phase. Simulation experiments conducted on 597 test sequences and 100 tracking sequences demonstrate that this method outperforms existing baselines. Real-world experiments on G1 robot across six task settings further validate its feasibility.
Critical strengths
-
This submission proposes a framework for generating mobile manipulation motions for humanoid robots from text. It specifically addresses the geometric constraints involved in object interaction, ensuring that the generated trajectories feature more plausible contact points and are better suited for physical execution.
-
The proposed method was comprehensively evaluated through both simulations and real-world experiments, which verified the feasibility of the generated interaction trajectories. The method performs better than baseline approaches.
-
The proposed method supports the enforcement of hard constraints on object pose without compromising the quality of the interaction.
Critical weaknesses
-
Although the proposed method yields high-quality motion generation, the scalability claimed in the paper remains unverified, both in terms of upstream data acquisition and downstream application. In the "Related Work" section, the authors describe CoLM as "providing a scalable data source for physical execution." However, there are no experiments analyzing the relationship between data scale and performance on the upstream side, nor is there any quantitative evaluation involving object categories or tasks outside the training set. On the downstream side, the generated data is not used to train or evaluate any general-purpose motion policies. Does this imply that the method is constrained by the existing motion-capture data of human-object interactions?
-
The downstream execution of this method requires training a separate policy for each reference, with each instance necessitating 3*10^8 environment steps of PPO training. Since the execution cost grows linearly with the number of tasks, downstream deployment is constrained. For instance, in real-world experiments, three distinct object-handling destinations actually correspond to three separately trained policies, which casts doubt on the method's practical deployability.
-
No variance is reported for the experiments: each test sequence is sampled only once during generation, and each reference is trained using only a single random seed during tracking. The mean and standard deviation, which would be derived from multiple random seeds (i.e., multiple sampling runs for generation and multiple training runs for tracking), are missing.
-
No video results were provided as supplemental.
Questions for authors
-
Could the authors discuss how the selection method and number of points sampled from the surfaces of objects within an interaction field affect trajectory generation? Furthermore, how do the authors handle objects with complex surface structures?
-
Could the authors evaluate the scalability of the generated downstream policies by testing them on a small set of general-purpose action strategies?
-
Could the authors discuss the possibility that the generated actions are primarily determined by object geometry and initial state, with text playing only a limited role? In the experiments, the training text itself is quite template-based, and the authors do not demonstrate how the generated results vary when only the prompt is changed while keeping the object and its initial state constant. It is difficult to assess the extent to which the text actually controls the actions.
3 comments
W3. Generation and Tracking Variance
We report the mean and standard deviation over three sampling seeds for generation and three PPO training seeds per fixed reference motion for tracking. We have updated this table in our revised manuscript.
| Method | FID ↓ | R@3 ↑ | MM-Dist ↓ | Diversity → | Contact F1 (%) ↑ | Foot Skate (cm/s) ↓ | Jitter (10² m/s³) ↓ | Joint Limit (%) ↓ | Tracking SR (%) ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Ground Truth | |||||||||
| DreamControl | |||||||||
| Kimodo | |||||||||
| CHOIS + Retarget | |||||||||
| CHOIS (G1) | |||||||||
| Ours (Human) + Retarget | |||||||||
| Ours w/o IF | |||||||||
| Ours w/o L_cons | |||||||||
| Ours w/o IF & L_cons | |||||||||
| Ours |
W4. Supplementary Videos
Videos of our generated motions, tracking rollouts, and real-world deployment are available on our anonymous project website. The website link is also included at the end of the abstract of our manuscript.
Q1. Object Surface Point Sampling
We use the same procedure for simple and complex shapes. We sample 24 points from the object surface by farthest-point sampling, which spreads the points across the surface more evenly than random sampling at the same budget. The generator uses a 1,024 point Basis Point Set (BPS) to encode the overall object shape. The 24 surface points capture local robot–object relationships in the interaction field. Surface points closer to each robot keypoint receive greater weight, so the field emphasizes the part of the object the robot is near.
We also conduct an ablation study on the number of sampled surface points:
| FPS Points | FID ↓ | Contact F1 (%) ↑ | Tracking SR (%) ↑ |
|---|---|---|---|
| 12 | 2.78 | 62.3 | 73.4 |
| 24 | 1.80 | 71.2 | 89.9 |
| 48 | 1.83 | 69.6 | 88.2 |
| 96 | 1.98 | 68.4 | 86.5 |
Using 12 points gives sparser surface coverage, which can leave local contact geometry underrepresented. Increasing the count to 24 improves all three metrics. Performance declines slightly at 48 and 96 points. This is because with only 15 body keypoints, adding more object points gives object–object relations greater weight in the interaction field, reducing the relative contribution of body–object relations. Among the four settings tested, 24 points give the best results on all three metrics.
Q3. Effect of the Text Prompt
To evaluate the effectiveness of text control, we keep the object and initial state fixed and only change the text prompt. We evaluate two settings: 1. Across actions: we compare prompts that instruct the robot to carry, push, or kick the same box 2. Within action: we compare four ways of asking the robot to carry the box.
Figures 12 and 13 in Appendix G.4 show that CoLM-Generator follows text and generates reasonable motion in both settings, showing the effectiveness of text control.
References
[1] Jiahao Ji, Ji Ma, Runhan Zhang, Runyi Yu, Wenjia Wang, Weiheng Chi, Qianqian Peng, Weichao Yan, Yongfei Gu, Ye Tian, Ting Wu, Longwei Li, Chun Yuan, Ruoli Dai, and Lei Han. HiPHI: A large-scale benchmark for high-precision human motion and object-interaction, 2026. URL https://arxiv.org/abs/2608.16222.
[2] Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, and Mengdi Xu. Weave: Learning whole-body dexterous loco-manipulation from human-object interactions, 2026. URL https://arxiv.org/abs/2609.16683.
[3] Sirui Chen, Shibo Zhao, Zhen Wu, Jiaman Li, Guanya Shi, and C. Karen Liu. SceneBot: Contact-prompted general humanoid whole body tracking with scene-interaction, 2026. URL https://arxiv.org/abs/2606.27581.
Dear Reviewer bnmo,
Thank you for your time and constructive feedback. We have responded to all the concerns raised in your review and provided additional experiments and clarifications.
As the discussion phase is approaching its end, we would appreciate any further feedback on whether our responses address your concerns. If any questions remain, we would be happy to clarify them during the remaining discussion period. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
We sincerely look forward to your feedback.
Thank you for your thoughtful review and constructive feedback. We have conducted additional experiments as advised, revised the manuscript and highlighted the changes in blue.
Please let us know if you have any further questions and we can provide any additional clarifications.
W1. Scalability and Out-of-Distribution Evaluation
We agree that scalability deserves evaluation at both the generation and execution stages.
W1.1 Data Coverage and Scalability
CoLM learns interaction priors from human–object motion-capture data, so the coverage of objects, motions, and contact modes in the training data affects its generation capabilities. Once trained, a single generator can synthesize new references from text, object geometry, and spatial constraints, reducing the need to collect a separate demonstration for each new spatial specification.
We have revised the relevant text in the Related Work section to clarify this point.
A systematic study of performance versus training-data scale remains future work, for which we plan to explore larger datasets such as one concurrent work HiPHI [1].
W1.2 Transfer to Unseen Tasks
We evaluated CoLM on 20 HiPHI [1] clips to test transfer to unseen tasks. Table 16 in Appendix G.2 of the revised manuscript reports joint-limit violations in the generated references and tracking success in simulation.
CoLM achieves 69.7% tracking success on these external sequences, providing additional evidence of cross-dataset transfer and the executability of the generated motions. The comparison with our original benchmark also quantifies the remaining generalization gap.
Figure 10 in Appendix G.2 presents four additional HiPHI examples spanning object lifting, carrying, and lowering, illustrating generated interactions beyond the original evaluation sequences.
W2 & Q2. General Interaction Tracking
We agree that training a separate policy for each generated clip limits execution scalability. CoLM primarily focuses on controllable loco-manipulation generation, and the tracking experiments are designed to assess the physical executability of the generated references. General interaction tracking, as explored in recent concurrent work such as Weave [2] and SceneBot [3], is a complementary research direction.
To directly address this concern, we trained a general tracker to follow multiple references without retraining between them. The tables compare tracking success and final object pose errors for the general tracker, per-clip CoLM, and per-clip HDMI. We have added these results to Table 15 in Appendix G.1 of the revised manuscript.
The general CoLM tracker outperforms per-clip HDMI while using one policy for multiple reference motions, with higher success rates in all five categories and lower final object pose errors on average, and with some performance degradation relative to per-clip CoLM-Tracker.
(a) Success rate (%) ↑
(b) Final object translation error (cm) ↓
(c) Final object orientation error (°) ↓