acceptodds
Under review as a conference paper at ICLR 2027

CoLM: Controllable Text-Driven Humanoid Loco-Manipulation Generation

Abstract

Humanoid loco-manipulation requires coordinated whole-body motion and accurate object interaction, yet collecting such demonstrations through teleoperation or motion capture is difficult to scale. Generative models provide a promising alternative, but language alone specifies only coarse task intent and offers limited control over object-level spatial goals. We present CoLM, a framework for generating and executing humanoid loco-manipulation from language and explicit object-level spatial constraints. Given a text description, object geometry, and the initial humanoid and object states, our model jointly generates humanoid and object motion while supporting spatial control ranging from a target object pose to a full trajectory. The generated motion is then executed by a motion tracking policy. We construct a humanoid loco-manipulation dataset by retargeting existing human-object interaction data to the Unitree G1 with contact-guided refinement. We further introduce the interaction field, a dense representation of humanoid-object spatial relations that serves as a shared interaction objective across generation and execution, being jointly predicted with motion during generation and used as a dense reward for tracking. Experiments show that our generator produces higher-quality motions than existing baselines and accurately satisfies object-level spatial constraints while preserving humanoid-object interaction, and our tracking policy achieves higher execution success. We further demonstrate the real-world applicability of CoLM through successful execution of diverse loco-manipulation tasks on the Unitree G1.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Loco-Manipulation, Robotic Data Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bnmoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission proposes CoLM, which takes text descriptions, object geometry, and the initial states of both the robot and the object as inputs to generate and execute manipulation tasks for a humanoid robot. It supports spatial control, ranging from target object poses to complete object trajectories. The method begins by retargeting six publicly available human-object interaction datasets for the Unitree G1 robot, recovering lost contact information through contact-guided refinement to construct a dataset of approximately 10,000 sequences. A diffusion model is then trained to jointly generate robot motion, object motion, and an "interaction field", a dense representation describing the relative geometric relationship between the robot and the object. Finally, a reinforcement learning-based tracking policy is employed to execute the generated actions. The interaction field serves both as a geometric consistency constraint during the generation phase and as a reward signal during the tracking phase. Simulation experiments conducted on 597 test sequences and 100 tracking sequences demonstrate that this method outperforms existing baselines. Real-world experiments on G1 robot across six task settings further validate its feasibility.

Critical strengths
  • This submission proposes a framework for generating mobile manipulation motions for humanoid robots from text. It specifically addresses the geometric constraints involved in object interaction, ensuring that the generated trajectories feature more plausible contact points and are better suited for physical execution.

  • The proposed method was comprehensively evaluated through both simulations and real-world experiments, which verified the feasibility of the generated interaction trajectories. The method performs better than baseline approaches.

  • The proposed method supports the enforcement of hard constraints on object pose without compromising the quality of the interaction.

Critical weaknesses
  • Although the proposed method yields high-quality motion generation, the scalability claimed in the paper remains unverified, both in terms of upstream data acquisition and downstream application. In the "Related Work" section, the authors describe CoLM as "providing a scalable data source for physical execution." However, there are no experiments analyzing the relationship between data scale and performance on the upstream side, nor is there any quantitative evaluation involving object categories or tasks outside the training set. On the downstream side, the generated data is not used to train or evaluate any general-purpose motion policies. Does this imply that the method is constrained by the existing motion-capture data of human-object interactions?

  • The downstream execution of this method requires training a separate policy for each reference, with each instance necessitating 3*10^8 environment steps of PPO training. Since the execution cost grows linearly with the number of tasks, downstream deployment is constrained. For instance, in real-world experiments, three distinct object-handling destinations actually correspond to three separately trained policies, which casts doubt on the method's practical deployability.

  • No variance is reported for the experiments: each test sequence is sampled only once during generation, and each reference is trained using only a single random seed during tracking. The mean and standard deviation, which would be derived from multiple random seeds (i.e., multiple sampling runs for generation and multiple training runs for tracking), are missing.

  • No video results were provided as supplemental.

Questions for authors
  • Could the authors discuss how the selection method and number of points sampled from the surfaces of objects within an interaction field affect trajectory generation? Furthermore, how do the authors handle objects with complex surface structures?

  • Could the authors evaluate the scalability of the generated downstream policies by testing them on a small set of general-purpose action strategies?

  • Could the authors discuss the possibility that the generated actions are primarily determined by object geometry and initial state, with text playing only a limited role? In the experiments, the training text itself is quite template-based, and the authors do not demonstrate how the generated results vary when only the prompt is changed while keeping the object and its initial state constant. It is difficult to assess the extent to which the text actually controls the actions.

3 comments

Thank you for your thoughtful review and constructive feedback. We have conducted additional experiments as advised, revised the manuscript and highlighted the changes in blue.

  • Text revision of scalability claim in Related Works section.
  • Experiment on generalization to unseen tasks on HiPHI dataset.
  • Experiment on general interaction tracking policy.
  • Experiment on variability across generation sampling seeds and PPO training seeds.
  • Experiment on Object surface sampling strategy.
  • Experiment on the effect of changing the text prompt.

Please let us know if you have any further questions and we can provide any additional clarifications.

W1. Scalability and Out-of-Distribution Evaluation

We agree that scalability deserves evaluation at both the generation and execution stages.

W1.1 Data Coverage and Scalability

CoLM learns interaction priors from human–object motion-capture data, so the coverage of objects, motions, and contact modes in the training data affects its generation capabilities. Once trained, a single generator can synthesize new references from text, object geometry, and spatial constraints, reducing the need to collect a separate demonstration for each new spatial specification.

We have revised the relevant text in the Related Work section to clarify this point.

A systematic study of performance versus training-data scale remains future work, for which we plan to explore larger datasets such as one concurrent work HiPHI [1].

W1.2 Transfer to Unseen Tasks

We evaluated CoLM on 20 HiPHI [1] clips to test transfer to unseen tasks. Table 16 in Appendix G.2 of the revised manuscript reports joint-limit violations in the generated references and tracking success in simulation.

CoLM achieves 69.7% tracking success on these external sequences, providing additional evidence of cross-dataset transfer and the executability of the generated motions. The comparison with our original benchmark also quantifies the remaining generalization gap.

SplitJoint Limit (%) ↓Tracking SR (%) ↑
Ours-test0.8289.9
HiPHI0.8369.7

Figure 10 in Appendix G.2 presents four additional HiPHI examples spanning object lifting, carrying, and lowering, illustrating generated interactions beyond the original evaluation sequences.


W2 & Q2. General Interaction Tracking

We agree that training a separate policy for each generated clip limits execution scalability. CoLM primarily focuses on controllable loco-manipulation generation, and the tracking experiments are designed to assess the physical executability of the generated references. General interaction tracking, as explored in recent concurrent work such as Weave [2] and SceneBot [3], is a complementary research direction.

To directly address this concern, we trained a general tracker to follow multiple references without retraining between them. The tables compare tracking success and final object pose errors for the general tracker, per-clip CoLM, and per-clip HDMI. We have added these results to Table 15 in Appendix G.1 of the revised manuscript.
The general CoLM tracker outperforms per-clip HDMI while using one policy for multiple reference motions, with higher success rates in all five categories and lower final object pose errors on average, and with some performance degradation relative to per-clip CoLM-Tracker.
(a) Success rate (%) ↑

MethodCarryPushPullKickSitAvg.
Ours (per-clip)90.591.088.790.389.089.9
Ours (general)84.590.382.986.481.485.1
HDMI (per-clip)81.887.480.782.781.182.7

(b) Final object translation error (cm) ↓

MethodCarryPushPullKickSitAvg.
Ours (per-clip)10.29.311.210.64.59.2
Ours (general)13.110.618.314.67.912.9
HDMI (per-clip)12.412.119.620.38.414.6

(c) Final object orientation error (°) ↓

MethodCarryPushPullKickSitAvg.
Ours (per-clip)9.48.710.79.63.98.5
Ours (general)16.79.312.313.47.811.9
HDMI (per-clip)14.310.316.118.29.013.6

W3. Generation and Tracking Variance

We report the mean and standard deviation over three sampling seeds for generation and three PPO training seeds per fixed reference motion for tracking. We have updated this table in our revised manuscript.

MethodFID ↓R@3 ↑MM-Dist ↓Diversity →Contact F1 (%) ↑Foot Skate (cm/s) ↓Jitter (10² m/s³) ↓Joint Limit (%) ↓Tracking SR (%) ↑
Ground Truth0.100.100.7090.7090.6900.6901.401.40100.0100.03.453.450.450.450.700.7091.491.4
DreamControl4.85±0.344.85^{±0.34}0.549±0.0050.549^{±0.005}0.760±0.0030.760^{±0.003}1.30±0.0041.30^{±0.004}28.9±1.728.9^{±1.7}9.03±1.29.03^{±1.2}1.54±0.121.54^{±0.12}1.73±0.041.73^{±0.04}35.3±3.235.3^{±3.2}
Kimodo3.82±0.173.82^{±0.17}0.661±0.0020.661^{±0.002}0.752±0.0020.752^{±0.002}1.31±0.0061.31^{±0.006}55.6±1.255.6^{±1.2}5.77±0.85.77^{±0.8}0.43±0.030.43^{±0.03}2.54±0.042.54^{±0.04}68.8±2.568.8^{±2.5}
CHOIS + Retarget3.52±0.203.52^{±0.20}0.578±0.0060.578^{±0.006}0.779±0.0030.779^{±0.003}1.34±0.0051.34^{±0.005}51.7±1.451.7^{±1.4}7.55±1.17.55^{±1.1}0.26±0.070.26^{±0.07}0.28±0.020.28^{±0.02}63.9±2.963.9^{±2.9}
CHOIS (G1)3.17±0.193.17^{±0.19}0.656±0.0050.656^{±0.005}0.734±0.0030.734^{±0.003}1.36±0.0071.36^{±0.007}48.9±1.248.9^{±1.2}8.17±0.88.17^{±0.8}0.60±0.090.60^{±0.09}0.95±0.040.95^{±0.04}59.8±2.759.8^{±2.7}
Ours (Human) + Retarget2.83±0.142.83^{±0.14}0.669±0.0040.669^{±0.004}0.702±0.0020.702^{±0.002}1.37±0.0041.37^{±0.004}69.0±1.169.0^{±1.1}4.10±0.64.10^{±0.6}0.19±0.050.19^{±0.05}0.35±0.030.35^{±0.03}84.7±2.584.7^{±2.5}
Ours w/o IF2.89±0.172.89^{±0.17}0.676±0.0070.676^{±0.007}0.718±0.0010.718^{±0.001}1.39±0.0051.39^{±0.005}63.2±1.263.2^{±1.2}4.45±0.54.45^{±0.5}0.38±0.030.38^{±0.03}0.86±0.050.86^{±0.05}75.0±2.175.0^{±2.1}
Ours w/o L_cons2.73±0.192.73^{±0.19}0.682±0.0050.682^{±0.005}0.705±0.0040.705^{±0.004}1.34±0.0041.34^{±0.004}62.3±0.762.3^{±0.7}4.29±0.74.29^{±0.7}0.36±0.060.36^{±0.06}0.81±0.040.81^{±0.04}76.8±2.676.8^{±2.6}
Ours w/o IF & L_cons3.28±0.163.28^{±0.16}0.665±0.0060.665^{±0.006}0.724±0.0050.724^{±0.005}1.35±0.0031.35^{±0.003}56.2±1.456.2^{±1.4}4.49±0.54.49^{±0.5}0.44±0.040.44^{±0.04}0.94±0.030.94^{±0.03}70.7±2.370.7^{±2.3}
Ours1.84±0.151.84^{±0.15}0.689±0.0060.689^{±0.006}0.697±0.0030.697^{±0.003}1.40±0.0031.40^{±0.003}71.7±1.171.7^{±1.1}4.37±0.44.37^{±0.4}0.41±0.030.41^{±0.03}0.82±0.030.82^{±0.03}89.6±2.189.6^{±2.1}

W4. Supplementary Videos

Videos of our generated motions, tracking rollouts, and real-world deployment are available on our anonymous project website. The website link is also included at the end of the abstract of our manuscript.


Q1. Object Surface Point Sampling

We use the same procedure for simple and complex shapes. We sample 24 points from the object surface by farthest-point sampling, which spreads the points across the surface more evenly than random sampling at the same budget. The generator uses a 1,024 point Basis Point Set (BPS) to encode the overall object shape. The 24 surface points capture local robot–object relationships in the interaction field. Surface points closer to each robot keypoint receive greater weight, so the field emphasizes the part of the object the robot is near.

We also conduct an ablation study on the number of sampled surface points:

FPS PointsFID ↓Contact F1 (%) ↑Tracking SR (%) ↑
122.7862.373.4
241.8071.289.9
481.8369.688.2
961.9868.486.5

Using 12 points gives sparser surface coverage, which can leave local contact geometry underrepresented. Increasing the count to 24 improves all three metrics. Performance declines slightly at 48 and 96 points. This is because with only 15 body keypoints, adding more object points gives object–object relations greater weight in the interaction field, reducing the relative contribution of body–object relations. Among the four settings tested, 24 points give the best results on all three metrics.


Q3. Effect of the Text Prompt

To evaluate the effectiveness of text control, we keep the object and initial state fixed and only change the text prompt. We evaluate two settings: 1. Across actions: we compare prompts that instruct the robot to carry, push, or kick the same box 2. Within action: we compare four ways of asking the robot to carry the box.

Figures 12 and 13 in Appendix G.4 show that CoLM-Generator follows text and generates reasonable motion in both settings, showing the effectiveness of text control.

References

[1] Jiahao Ji, Ji Ma, Runhan Zhang, Runyi Yu, Wenjia Wang, Weiheng Chi, Qianqian Peng, Weichao Yan, Yongfei Gu, Ye Tian, Ting Wu, Longwei Li, Chun Yuan, Ruoli Dai, and Lei Han. HiPHI: A large-scale benchmark for high-precision human motion and object-interaction, 2026. URL https://arxiv.org/abs/2608.16222.

[2] Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, and Mengdi Xu. Weave: Learning whole-body dexterous loco-manipulation from human-object interactions, 2026. URL https://arxiv.org/abs/2609.16683.

[3] Sirui Chen, Shibo Zhao, Zhen Wu, Jiaman Li, Guanya Shi, and C. Karen Liu. SceneBot: Contact-prompted general humanoid whole body tracking with scene-interaction, 2026. URL https://arxiv.org/abs/2606.27581.

Dear Reviewer bnmo,

Thank you for your time and constructive feedback. We have responded to all the concerns raised in your review and provided additional experiments and clarifications.

As the discussion phase is approaching its end, we would appreciate any further feedback on whether our responses address your concerns. If any questions remain, we would be happy to clarify them during the remaining discussion period. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_jR3MOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoLM integrates a contact-guided retargeting technique from six human-object interaction datasets, a text- and object-condition-based humanoid robot-object motion generator, and a physics-based tracker. A dense “interaction field” represents the spatial relationships between the body and objects during generation and tracking. It accepts object target poses, waypoints, or complete trajectories as input. This paper evaluates 597 test sequences for the generation task, 100 generated reference sequences for the tracking task, and conducts 120 physical experiments across four Unitree G1 tasks.

Critical strengths
  1. The paper covers the full pipeline from text and object-level constraints to physical humanoid execution. Jointly generating body and object motion is relevant to loco-manipulation and distinguishes it from body-only approaches.

  2. Contact-guided retargeting improves Contact F1 to 82.5% on the 9,130 clips where all methods return feasible solutions, compared with 43.1% for OmniRetarget and 28.8% for GMR. It also yields lower reported jitter.

  3. The ablations support the contribution of the interaction field. Removing both field prediction and geometric consistency reduces Contact F1 from 71.2% to 56.0% and simulated tracking success from 89.9% to 71.1%. Removing the tracker’s interaction reward reduces success to 83.2%.

  4. The paper includes hardware results rather than relying solely on simulation: 111 of 120 trials succeed across box carrying, tripod carrying, box kicking, and sitting.

Critical weaknesses
  1. The execution pipeline does not scale easily to new requests. The tracker trains a separate policy for each reference motion, which is a substantial cost for a system presented as text-driven. The paper should report the time required to turn a new prompt into an executable policy and test whether one policy can track multiple generated motions.

  2. The spatial-control results mainly measure generated references, not executed motions. The zero errors in Table 4 follow from imposing the requested constraints during generation. In simulation, the tracked motion still has mean final object errors of 9.2 cm and 8.5° (Sec. 4.3). End-to-end accuracy should be reported separately for target poses, waypoints, and full trajectories.

  3. The hardware evaluation is narrow. The 120 trials repeat six settings across four task types. Testing unseen physical object instances is useful, but the paper also acknowledges degraded generation for unseen object and motion categories (Sec. 5). The current results do not establish that the system handles substantially new tasks.

  4. Some physical-quality metrics remain below the reference or a baseline. Foot skating is 4.34 cm/s versus 3.45 cm/s for ground truth, and joint-limit violations are 0.82% versus 0.28% for CHOIS + Retarget. The paper should show where these errors occur and whether they contribute to tracking failures.

Questions for authors
  • What happens when a waypoint or full trajectory is outside reach, collides with the scene, or requires a new contact mode?
  • Can one tracker policy follow multiple generated references without retraining, and how does its success compare with the per-clip policies?
5 comments

Thank you for your thoughtful review and constructive feedback. We have conducted additional experiments as advised, revised the manuscript and highlighted the changes in blue.

  • Experiment on general interaction tracking policy.
  • Experiment on end-to-end spatial control accuracy.
  • Experiment on the feasibility of commands involving reachability, obstacles, or new contact modes.
  • Experiment on generalization to unseen tasks on HiPHI dataset.
  • Experiment on how foot skating and joint-limit violations relate to tracking failures.

Please let us know if you have any further questions and we can provide any additional clarifications.

W1 & Q2. Execution Cost and General Tracker

W1.1 Per-Clip Policy Training Cost

We agree that training a separate policy for each generated clip limits execution scalability. CoLM primarily focuses on controllable loco-manipulation generation, and the tracking experiments are designed to assess the physical executability of the generated references. General interaction tracking, as explored in recent concurrent work such as Weave [1] and SceneBot [2], is a complementary research direction. In the original setup, PPO training takes about two hours per clip on a single RTX 6000 Ada GPU. This cost could potentially be reduced by training a residual policy on top of a pretrained general motion tracker, following approaches such as ResMimic [3].

W1.2 General Policy Evaluation

We trained a general tracker to follow multiple references without retraining between them. The tables compare tracking success and final object pose errors for the general tracker, per-clip CoLM, and per-clip HDMI. We have added these results to Table 15 in Appendix G.1 of the revised manuscript.
The general CoLM tracker outperforms per-clip HDMI while using one policy for multiple reference motions, with higher success rates in all five categories and lower final object pose errors on average, and with some performance degradation relative to per-clip CoLM.

(a) Success rate (%) ↑

MethodCarryPushPullKickSitAvg.
Ours (per-clip)90.591.088.790.389.089.9
Ours (general)84.590.382.986.481.485.1
HDMI (per-clip)81.887.480.782.781.182.7

(b) Final object translation error (cm) ↓

MethodCarryPushPullKickSitAvg.
Ours (per-clip)10.29.311.210.64.59.2
Ours (general)13.110.618.314.67.912.9
HDMI (per-clip)12.412.119.620.38.414.6

(c) Final object orientation error (°) ↓

MethodCarryPushPullKickSitAvg.
Ours (per-clip)9.48.710.79.63.98.5
Ours (general)16.79.312.313.47.811.9
HDMI (per-clip)14.310.316.118.29.013.6

W2. End-to-End Spatial Control Accuracy

We report End-to-end spatial control accuracy results as below:

MethodControl modeTobjendT^{end}_{obj} (cm) ↓OobjendO^{end}_{obj} (°) ↓TobjT_{obj} (cm) ↓OobjO_{obj} (°) ↓Tobj,execendT^{end}_{obj,exec} (cm) ↓Oobj,execendO^{end}_{obj,exec} (°) ↓
Ground Truth—00008.78.8
CHOIS + Retargetwaypoints17.3746.7020.8041.2319.2442.30
CHOIS (G1)waypoints8.1212.1519.4521.5712.4315.32
Ourstarget0.000.0024.0732.009.18.8
Ourswaypoints0.000.0019.3924.9810.29.2
Oursfull trajectory0.000.000.000.0010.810.3

CoLM achieves final object translation and rotation errors of 9.1 cm and 8.8° for target poses, 10.2 cm and 9.2° for waypoints, and 10.8 cm and 10.3° for full trajectories. It has the lowest execution error on both metrics among the generation methods in each mode, remaining close to the errors obtained by tracking ground-truth references (8.7 cm and 8.8°).

W3. Out-of-Distribution Evaluation

To test transfer to unseen tasks, we generated motion references for 20 HiPHI [4] clips. Table 16 in Appendix G.2 of the revised manuscript reports joint-limit violations in the generated references and tracking success in simulation.

CoLM achieves 69.7% tracking success on these external sequences, providing additional evidence of cross-dataset transfer and the executability of the generated motions. The comparison with our original benchmark also quantifies the remaining generalization gap.

SplitJoint Limit (%) ↓Tracking SR (%) ↑
Ours-test0.8289.9
HiPHI0.8369.7

Figure 10 in Appendix G.2 presents four additional HiPHI examples spanning object lifting, carrying, and lowering, illustrating generated interactions beyond the original evaluation sequences.


W4. Foot Skating and Joint-Limit Violations

W4.1 Where These Errors Occur

CoLM supervises foot-contact indicators but does not explicitly penalize foot sliding during contact. We mainly observe residual skating during fast-walking segments. The generated motions exhibit slightly more foot skating than the ground-truth motions (4.34 vs. 3.45 cm/s).

Joint-limit violations tend to occur during kinematically demanding interactions, such as rapid object manipulation or handling large objects. CoLM directly generates robot motion without joint-limit clipping at inference, whereas CHOIS + Retarget additionally applies joint-limit-constrained retargeting to each generated motion, helping explain its lower violation rate.

W4.2 Relationship to Tracking Failures

We measured Spearman’s rank correlation between each metric of the generated references and the corresponding per-clip tracking failure rate (1 − tracking success rate).

Reference MetricSpearman’s ρ95% CI
Foot Skate+0.12
−0.08,+0.23−0.08, +0.23

| | Joint Limit | +0.05 |

−0.12,+0.16−0.12, +0.16

|

Both correlations are weak, and both confidence intervals include zero. Thus, we find no clear evidence of a monotonic association between either metric and tracking failure rates on the evaluated clips.

One possible explanation is that foot skating is limited and joint-limit violations are infrequent in the generated references(4.34cm/s and 0.82%), so the tracker can often tolerate these deviations.


Q1. Feasibility of Spatial Commands

Q1.1 Unreachable Commands

CoLM enforces the commanded object poses and generates the corresponding whole-body motion. If a waypoint is unreachable at the specified time, the generated object reference can still reach it, while the robot may lose contact. Revising the waypoint or allowing more time for the robot to approach can make the command feasible.

Q1.2 Scene Collisions

For scene collisions, we tested an upstream geometric planner that supplies a detour around an obstacle (Figure 11 in Appendix G.3). CoLM followed the original path through the obstacle in all three samples. With the detour, the box cleared the obstacle in all three samples and the robot cleared it in two. This shows that CoLM can be combined with a high-level planner to avoid collision.

Q1.3 New Contact Modes

If a trajectory requires a contact mode unseen during training, CoLM can still generate a candidate motion from the prompt, object geometry, and trajectory. The interaction field encourages consistent robot–object contact, but the body may fail to establish or maintain the contact needed for the new action. As reported in W3, CoLM achieves 69.7% tracking success on 20 unseen HiPHI clips involving unseen contact modes, compared with 89.9% on our original test set. This shows that CoLM can transfer to some unseen contact modes, with a remaining success rate gap.

References

[1] Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, and Mengdi Xu. Weave: Learning whole-body dexterous loco-manipulation from human-object interactions, 2026. URL https://arxiv.org/abs/2609.16683.

[2] Sirui Chen, Shibo Zhao, Zhen Wu, Jiaman Li, Guanya Shi, and C. Karen Liu. SceneBot: Contact-prompted general humanoid whole body tracking with scene-interaction, 2026. URL https://arxiv.org/abs/2606.27581.

[3] Siheng Zhao, Yanjie Ze, Yue Wang, C. Karen Liu, Pieter Abbeel, Guanya Shi, Rocky Duan. ResMimic: From General Motion Tracking to Humanoid Whole-Body Loco-Manipulation via Residual Learning. 2025. URL https://arxiv.org/abs/2510.05070.

[4] Jiahao Ji, Ji Ma, Runhan Zhang, Runyi Yu, Wenjia Wang, Weiheng Chi, Qianqian Peng, Weichao Yan, Yongfei Gu, Ye Tian, Ting Wu, Longwei Li, Chun Yuan, Ruoli Dai, and Lei Han. HiPHI: A large-scale benchmark for high-precision human motion and object-interaction, 2026. URL https://arxiv.org/abs/2608.16222.

Reviewer_jR3MOpenReview ↗

Response by Reviewer jR3M

Thank you for the thorough response and the additional experiments. The general tracker results and the foot-skating and joint-limit analysis address my concerns well. A few points would still benefit from clarification.

For the general tracker, were the evaluation references held out from its training set? If the policy is trained on the same clips it is evaluated on, it shows that one policy can track several references, but not that newly generated motions can be executed without additional training, which is the scalability question.

For HiPHI, how were the text prompts and spatial controls obtained for the 20 clips? Also, the response to Q1.3 cites this experiment as evidence of transfer to unseen contact modes, but the examples in Figure 10 (lifting, carrying, lowering) appear to overlap with the training categories. Which of these clips involve contact modes not seen in training?

Finally, the answers on unreachable waypoints and obstacles suggest that CoLM can produce references where the object reaches the commanded pose while the robot loses contact, and that it follows paths through obstacles unless an external planner intervenes. I think these should be stated explicitly as limitations in the paper.

Overall, the rebuttal addresses most of my concerns, and I will maintain my score in support of acceptance.

Reviewer_2N4POpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CoLM, a framework for generating and executing humanoid loco-manipulation motions. It first retargets existing human-object interaction data to the Unitree G1 using GMR with contact-guided refinement. A diffusion model then jointly generates humanoid motions, object trajectories, and an Interaction Field (IF), which explicitly models robot-object spatial relationships to improve contact consistency. Finally, a PPO-based motion tracking policy is trained to execute the generated motions on a real humanoid robot.

Critical strengths

(1) Effective Method Design. The paper integrates the Interaction Field (IF) into both diffusion-based motion generation and PPO-based tracking, providing a unified representation of robot-object interaction geometry. Ablation studies demonstrate its effectiveness in improving contact consistency and tracking performance.

(2) Comprehensive Experimental Validation. The paper provides thorough evaluations covering motion retargeting, motion generation, spatial control, simulation tracking, and real-world deployment. In particular, the 120 real-world trials provide convincing evidence of the framework's practical effectiveness.

Critical weaknesses

(1) Limited Methodological Novelty. Although the proposed method is effective, its novelty is somewhat limited, with the main contribution lying in the integration of existing techniques. In particular, the Interaction Field is conceptually similar to the interaction-aware geometric representation used in OmniRetarget. While applying this representation to diffusion-based generation and RL tracking is valuable, the overall contribution is more incremental than fundamentally novel.

(2) Limited Scalability of Motion Tracking. Each generated reference motion requires a separately trained tracking policy, making the execution pipeline computationally expensive and less practical for new tasks. Although motion tracking is not the main focus of this paper, this limitation reduces the framework's practical scalability.

Questions for authors

The effectiveness of the Interaction Field (IF) has been demonstrated in the experiments. However, could the authors further explain why this particular representation was chosen over simpler geometric representations, such as relative keypoint positions or contact distances? A more convincing explanation or comparison would help better establish the methodological contribution of IF.

4 comments

Thank you for your thoughtful review and constructive feedback. We have conducted additional experiments as advised, revised the manuscript and highlighted the changes in blue.

  • Experiment on general interaction tracking policy.
  • Experiment on how the Interaction Field compares with relative keypoint positions and contact distances.

Please let us know if you have any further questions and we can provide any additional clarifications.

W1. Methodological Novelty

We agree that our Interaction Field is conceptually similar to OmniRetarget’s representation of interaction geometry (for retargeting). We instead use it to supervise robot–object geometric consistency during motion generation and to reward preservation of this geometry during tracking policy training.

Our main methodological contribution is an interaction-aware diffusion formulation for controllable humanoid loco-manipulation. It jointly denoises robot motion, object motion, and interaction geometry through cross-attending streams, with contact-weighted geometric consistency supervision computed through forward kinematics. Prescribed object poses are restored at every denoising step to enforce strict spatial constraints in the generated reference. To support this formulation, we also develop contact-guided retargeting and construct a high-quality humanoid loco-manipulation dataset.


W2. Scalability of Motion Tracking

We agree that training a separate policy for each generated clip limits execution scalability. CoLM primarily focuses on controllable loco-manipulation generation, and the tracking experiments are designed to assess the physical executability of the generated references. General interaction tracking, as explored in recent concurrent work such as Weave [1] and SceneBot [2], is a complementary research direction.

To directly address this concern, we trained a general tracker to follow multiple references without retraining between them. The tables compare tracking success and final object pose errors for the general tracker, per-clip CoLM, and per-clip HDMI. We have added these results to Table 15 in Appendix G.1 of the revised manuscript.
The general CoLM tracker outperforms per-clip HDMI while using one policy for multiple reference motions, with higher success rates in all five categories and lower final object pose errors on average, and with some performance degradation relative to per-clip CoLM-Tracker.

(a) Success rate (%) ↑

MethodCarryPushPullKickSitAvg.
Ours (per-clip)90.591.088.790.389.089.9
Ours (general)84.590.382.986.481.485.1
HDMI (per-clip)81.887.480.782.781.182.7

(b) Final object translation error (cm) ↓

MethodCarryPushPullKickSitAvg.
Ours (per-clip)10.29.311.210.64.59.2
Ours (general)13.110.618.314.67.912.9
HDMI (per-clip)12.412.119.620.38.414.6

(c) Final object orientation error (°) ↓

MethodCarryPushPullKickSitAvg.
Ours (per-clip)9.48.710.79.63.98.5
Ours (general)16.79.312.313.47.811.9
HDMI (per-clip)14.310.316.118.29.013.6

References

[1] Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, and Mengdi Xu. Weave: Learning whole-body dexterous loco-manipulation from human-object interactions, 2026. URL https://arxiv.org/abs/2609.16683.

[2] Sirui Chen, Shibo Zhao, Zhen Wu, Jiaman Li, Guanya Shi, and C. Karen Liu. SceneBot: Contact-prompted general humanoid whole body tracking with scene-interaction, 2026. URL https://arxiv.org/abs/2606.27581.

Q1. Why the Interaction Field?

Relative keypoint position describes where a body keypoint lies in the object’s coordinate frame, ignoring the object’s geometry. Contact distances measure how close each keypoint is to the surface, leaving the location of the closest surface point and the direction to it unspecified. As a result, different robot–object interaction can have identical relative positions or contact distances, making the intended interaction ambiguous. The Interaction Field explicitly combines body keypoints and object surface samples through distance-weighted neighborhoods, emphasizing local interaction geometry and the arrangement of nearby body parts.

We conduct experiments on these three representations. All these methods share the same architecture but with different geometric representation.

Geometric representationFIDR@3MM-DistContact F1Foot SkateJitterJoint LimitTracking SR
Relative keypoint positions2.940.6420.70854.34.360.420.8969.4
Contact distances2.730.6350.71260.14.400.390.9372.3
Ours w/o IF2.920.6760.71563.24.420.400.8475.3
Ours1.800.6870.69771.24.340.410.8289.9

Results show that the Interaction Field outperforms both baselines on most metrics, with the largest gains in FID, Contact F1, and Tracking SR. By emphasizing the relationships between nearby body keypoints and object surface points, the IF helps preserve the intended interaction in generated motions and makes them easier to track.

Dear Reviewer 2N4P,

Thank you for your time and constructive feedback. We have responded to all the concerns raised in your review and provided additional experiments and clarifications.

As the discussion phase is approaching its end, we would appreciate any further feedback on whether our responses address your concerns. If any questions remain, we would be happy to clarify them during the remaining discussion period. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Thank you for the detailed response and additional experiments. I acknowledge that applying the IF formulation to geometric consistency constraints during interaction motion generation and interaction rewards in subsequent PPO tracking introduces a certain degree of novelty. However, fundamentally, it still shares strong similarities with the existing representation in OmniRetarget. In addition, the supplementary experiments further reveal that the method still has limitations in generalization, with a noticeable drop in execution success rate on the HiPHI dataset.

Overall, I believe that the current method has not fully addressed the aforementioned issues and still has room for further improvement. Therefore, I have decided to maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript and Response to All Reviewers

We sincerely thank all the reviewers for their thoughtful reviews and constructive feedback. We appreciate the strengths they highlighted across the paper, including the design of the Interaction Field in both generation and tracking and the breadth of the evaluation (2N4P), contact-guided retargeting and joint robot–object motion generation with hardware deployment (jR3M), and precise object pose control that preserves interaction quality (bnmo).

In response to the reviewers’ feedback, we conducted additional experiments and revised the manuscript, with changes highlighted in blue color. The main updates are summarized below.

  1. Interaction Field Design. We compare the Interaction Field with relative keypoint positions and contact distances under the same architecture. We also vary the strategy of object surface points sampling to evaluate its effect on generation and tracking.
  2. General tracker. We train one general tracking policy on multiple reference motions and compare it with per-clip policies.
  3. Cross-Dataset Evaluation. We evaluate CoLM on external interaction sequences from HiPHI dataset and assess the physical executability of the generated references.
  4. Generation and Tracking Variability. We report means and standard deviations over three sampling seeds for generation and three PPO training seeds per reference motion for tracking.
  5. Spatial and text control. We measure end-to-end spatial control accuracy. We evaluate CoLM in combination with a high-level planner for obstacle avoidance.. We also assess the effect of text on the generated motion.

We have addressed the queries and points raised by each reviewer below and made replies for clarification. Please let us know if you have any further questions and we can provide any additional clarifications.

We thank the reviewers again for their time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.