acceptodds
Under review as a conference paper at ICLR 2027

Zero-HLM: Generalizable Humanoid Loco-Manipulation without Real Robot Data

Abstract

Learning generalizable humanoid loco-manipulation remains bottlenecked by costly real-world demonstrations. Simulation offers a scalable alternative, but extending it to humanoids requires solving two coupled challenges: constructing scene-scale environments that are both visually faithful and physics-ready, and generating successful demonstrations under unified whole-body control. We present Zero-HLM, a real-to-sim-to-real framework that transforms smartphone captures and simulated source demonstrations into scalable training data for humanoid loco-manipulation. We couple Gaussian visual reconstruction with agentic physical scene construction to build interaction-ready digital twins with aligned visual and physical geometry. Within the environment, our motion–environment co-adaptation strategy synthesizes unified whole-body references and uses rollout feedback to restore interaction geometry when tracking deviations disrupt execution. We successfully generate 4,800 physically validated demonstrations in simulation using Zero-HLM, averaging a 1.8× speedup over real-world data collection. Policies trained solely on these data transfer zero-shot to a Unitree G1 across three long-horizon tasks. With GR00T N1.7, simulation-only training achieves 66.7% mean full-task success with an absolute 20% gain over real-data training under matched configurations. Across five generalization settings, pooled real-world success increases from 23% to 69% without additional real demonstrations. To support systematic evaluation, we establish a benchmark with nine skill stages and shared protocols for comparing models and studying data scaling.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Loco-Manipulation, Simulation Data Generation, Zero-Shot Sim-to-Real

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_813nOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper investigates how simulation-generated training data can reduce the reliance on real-world demonstrations for humanoid mobile manipulation. The proposed Zero-HLM framework combines Gaussian-based scene reconstruction, agent-assisted physical modeling, and whole-body motion generation from source demonstrations. It jointly adjusts reference motions and environment configurations based on execution feedback to reduce the impact of motion-tracking errors on interaction success. The generated data is used to train visuomotor policies. The authors evaluate the approach through real-world deployment across multiple mobile manipulation tasks, component ablations, and simulation benchmarks. The main contribution is a unified pipeline that integrates scene construction and whole-body interaction data generation, with demonstrated transfer to real robots. The central technical claim is that execution-aware joint adaptation of motions and environments can improve the effectiveness of training data generation.

Critical strengths

1. Provides meaningful evidence of real-world transfer.
The paper uses generated data to train mobile manipulation policies for tasks involving walking, low-level grasping, bimanual carrying, and foot contact. It demonstrates real-world deployment without using real demonstrations of the target tasks. The comparable performance to training with the same number of real demonstrations also supports the practical value of the generated data. This is the strongest evidence in favor of the paper.

2. Addresses practical interaction issues in whole-body motion tracking.
The authors recognize that kinematically valid reference motions do not always preserve the required hand–object relationships during execution. They propose corrections for different types of errors. The ablation results suggest that these corrections improve the data generation success rate within the proposed pipeline, showing practical engineering value.

Critical weaknesses

1. The independent contribution of execution-aware correction is not sufficiently verified.
The current ablations mainly report data generation success rates, while environment correction also changes object or target positions. Therefore, higher success rates may reflect a more effective generation process, but may also come with changes in the distribution of generated configurations. The paper does not sufficiently report target coverage after correction. It also lacks controlled baselines using the same source demonstrations, controller, and renderer to evaluate downstream benefits under equal coverage or generation budgets. In addition, it is unclear whether the full trajectory is physically revalidated after modifying the environment. These issues make it difficult to assess how much this mechanism improves the generation of useful training data for the target tasks.

2. The generalization experiments do not clearly separate expanded training coverage from performance on unseen configurations.
The main comparisons change both the amount of training data and the coverage of configurations. The appendix shows that the expanded positions and alternative object–target combinations are already included in the corresponding simulation training data. Thus, the results mainly support sim-to-real transfer after targeted simulation training. This is valuable, but does not establish generalization to configurations not covered during training. Task C, which best reflects leg-based interaction, is only evaluated under the BASE setting, and some target positions are coupled with the robot's final position. The lack of equal-data coverage comparisons and held-out configuration tests limits the evidence for the claimed generalization ability.

Questions for authors

A. Key Questions That May Affect My Evaluation

1. Method clarification: What is the full validation process after environment correction?
After applying Eqs. (3) and (4), is the entire trajectory re-executed from the modified initial scene? Please clarify the order of correction, replay, and acceptance checks. Can the final accepted trajectories be replayed successfully without further changes to the environment state? This directly affects my assessment of the physical validity of the generated data.

2. Key comparison: What benefits does the correction mechanism provide while maintaining target coverage?
Could the authors compare the full method with a demonstration transformation baseline that does not modify the environment, using the same source demonstrations, SONIC controller, and visual renderer? Please report the final configuration coverage and downstream real-world performance. The existing ablations have already collected successful trajectories for each variant, which could be used for this comparison. Showing that the full method reduces generation costs while maintaining coverage, or improves real-world performance under the same budget, would strengthen the contribution.

3. Generalization: How much do expanded coverage and compositional generalization contribute under equal data sizes?
Could the authors compare models trained on equal amounts of BASE and EXPANDED simulation data, evaluated on the same real-world configurations? In addition, could some "robot start position × manipulation target position" combinations be held out? Each factor should appear separately during training, but their combination should remain unseen. This would help evaluate compositional generalization and better clarify the scope of the generalization claims.

B. Minor Clarification: Statistical Reporting and Raw Counts in Data Generation Experiments

The following concerns are reporting details that I suggest checking. They are not reasons for rejection and do not affect my final score.

Table 5 on page 7 reports a 61% generation success rate for the full method on Task A under the BASE setting, and states that 100 successful trajectories were collected for each variant. However, Appendix B.4.1 and Figure 13 report a 33% generation success rate for Task A. These numbers may come from different configuration sets, experimental runs, or evaluation stages, but their relationship is not clearly explained. Please clarify the evaluation scope, number of successes, and total number of attempts for each result. Also, please specify whether retries after correction are counted as new attempts.

In addition, Figure 14 reports failure category percentages of 87% and 13% for Sim-Astra on Task C, while stating that the total number of attempts is unavailable. Please explain how these percentages were calculated and whether the corresponding category counts or summary records are available. It would also help to clarify how these statistics relate to the timing experiments in Table 2, including how failures and retries are counted in the reported time.

The current information is insufficient to conclude that these differences are numerical inconsistencies. However, there may be reporting errors, and I encourage the authors to check them. For now, these concerns are not reasons for rejection and do not affect my final score.

Reviewer_1qVkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Zero-HLM generates humanoid training data in reconstructed simulation scenes. It adapts demonstrations and adjusts scene configurations to obtain successful rollouts. Policies trained on these data transfer to a Unitree G1 on three tasks.

Critical strengths

s1. I appreciate the real-robot evaluation, especially the tasks combining walking, carrying, and foot interaction.

s2. The results suggest that generated data can be a useful substitute for real demonstrations, with further gains from scaling up.

Critical weaknesses

w1. I am concerned that moving objects to accommodate execution errors could make the sampled tasks easier. It is unclear how much of the intended variation remains after correction.

w2. The comparison with a general-purpose agent is useful, but does not show how the method compares with existing humanoid data-generation pipelines.

w3. The generalization results are narrower than they first appear: the new configurations are covered in simulation training, and evaluation stays within one reconstructed scene.

w4. The generation speedup is encouraging, but the total cost remains unclear once setup and source demonstrations are included.

Questions for authors

q1. How much do objects move during correction? Do you replay each final trajectory from the beginning with the corrected scene fixed?

q2. Could you compare with an existing humanoid data-generation method using the same demonstrations and budget?

q3. How well does the policy handle configurations not included in simulation training?

q4. How much time and human effort does the full pipeline require, including reconstruction, calibration, and source demonstrations?

1 comment
Reviewer_813nOpenReview ↗

Discussion with Reviewer 1qVk: How the Revised Evidence Addresses the Main Concerns

Thank you for raising the questions about configuration correction, comparisons with existing methods, generalization, and the complete collection cost. These concerns overlap substantially with my initial reservations. After reviewing the authors’ responses and the revised manuscript, I found the additional evidence persuasive and have raised my rating to 4. I would like to highlight the changes that most affected my assessment.

Regarding whether moving objects simplifies the sampled tasks, Appendix B.4.4 now reports both correction magnitudes and the spatial coverage retained after correction. For targets requiring adjustment, median translations are 1.5–4.1 cm, while the reported robot-start and object-position coverage retention ranges from 77.4% to 100%. Together with complete physical re-execution and the new downstream ablations at equal demonstration counts, these results substantially alleviate my concern that the procedure succeeds primarily by concentrating the data into a narrow set of easier configurations. The revised evaluation now examines the consequences of correction for both coverage and policy learning.

The added LEGS comparison is another consequential improvement. The adapted baseline shares the source demonstrations, SONIC controller, reconstructed scene, and Gaussian renderer, while retaining its procedural motion-primitive generation approach. With 100 accepted demonstrations per task, Zero-HLM achieves full-task success of 7/20 and 10/20 on Tasks A and B, respectively, compared with 2/20 for LEGS on each task. It also requires less time to collect a fixed number of successful demonstrations. This directly addresses the absence of a specialized humanoid data-generation baseline and provides stronger evidence for the proposed approach beyond the shared controller and rendering infrastructure.

For generalization, the new held-out combination experiment is particularly relevant. Every individual robot-start and object position appears during training, but four combinations are excluded, including configurations after correction. The real-world results therefore provide evidence of compositional generalization beyond direct training coverage of the evaluated combinations. The evaluation remains within one reconstructed scene, and cross-scene generalization is appropriately left as future work. Within the paper’s stated scope, however, the evidence is materially stronger than in the original submission.

Appendix F also makes the cost discussion more informative by including reconstruction, configuration, calibration, and source-demonstration collection. The authors report 27.14 hours for the complete pipeline producing 2,400 successful demonstrations, with 53 minutes of active human involvement. They explicitly identify the large-scale real-teleoperation comparison as an extrapolation from smaller-scale measurements. This makes the practical benefit clearer: the initial preparation and source demonstrations can be reused to expand the dataset without repeatedly incurring human teleoperation effort.

Taken together, these additions substantially address the reasons that initially led me to question acceptance. The real-robot results, combined with the more informative controls and transparent accounting, now support a positive assessment in my view. I hope these findings are useful when reconsidering the revised submission, and I would welcome further discussion of any specific issue that remains consequential to your recommendation.

Reviewer_VFthOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Zero-HLM generates humanoid loco-manipulation demonstrations using Gaussian scene reconstruction, agentic physical modeling, source-motion adaptation, and execution-conditioned correction. Successful simulated rollouts provide egocentric observations and SONIC latent actions plus hand commands for policy training. The authors report 4,800 demonstrations and real G1 deployment on three multi-stage tasks. With 800 simulation demonstrations per task, mean BASE success is 66.7%, versus 46.7% with 100 real demonstrations. The matched-count simulation result is 43.3%, and expanded synthetic coverage yields 69/100 successes on shifted configurations and an object–goal swap, versus 23/100 for the real-data baseline.

Critical strengths
  • The execution-conditioned correction addresses a concrete difficulty in humanoid data generation: balanced motion tracking can still produce incorrect hand–object alignment. The method compensates for root, planar interaction, and grasp-height errors, and removing the robot SE(2) correction alone drops generation yield from 61% to 16%.
  • The tasks go beyond arm manipulation, including squat-picking and closing a cabinet door with the foot while carrying a basket. Task C transfers to the real robot, with the kick stage at 19/20 and full-task success at 13/20, compared with 9/20 for real data.
  • The mesh vs. Gaussian comparison is a useful data point for real-to-sim work on mobile robots. Changing only the scene rendering raises real-world pick success on Task A from 3/20 to 10/20.
Critical weaknesses
  • The headline improvement over real data is confounded by dataset size. Table 6 gets the 20-point advantage by comparing 800 simulation demonstrations per task with 100 real ones, while at the matched count simulation reaches 43.3%, slightly below 46.7% for real data.
  • The generalization result mainly reflects data coverage. Sim-Ours(800)* is generated around the EXPANDED test configurations, including 400 bitter-melon-to-plate demos, while the real-data policy never sees these settings. So 69% vs. 23% does not test generalization beyond the generated configurations.
  • Execution-conditioned correction moves scene assets to the executed root pose (Eq. 3) and shifts objects by the extra tracking error relative to a successful source rollout (Eq. 4), rather than changing the robot's motion. The kept demos may therefore never show recovery from a bad walking endpoint. Tab. 5 reports only generation yield, not policy success.
  • The data-generation side has limited novelty and is not compared with existing generation schemes. Eq. (1) follows MimicGen, and locomotion references and tracking come from SONIC. Prior humanoid pipelines such as DemoHLM and HumanoidMimicGen are only discussed in Sec. 2 and Tab. 1, and the only experimental baseline is GPT-6 Astra, which needed human-assisted debugging.
  • The co-training result is not well explained. Adding 100 real demos to 800 sim demos lowers mean success from 66.7% to 53.3%, which the authors attribute to the visual gap. However, Mix-50 in Tab. 10 also mixes sim and real images and succeeds in both regions, where each source alone fails in one.
Questions for authors
  • How exactly is the SE(2) correction applied? Is the rollout re-run from t=0 with the moved assets, and do moved assets that also appear in the Gaussian background create duplicated or misaligned visuals?
  • What is the real-world success of Sim-Ours(800) trained on BASE only and evaluated on the EXPANDED settings?
  • Why does adding real data hurt Sim(800)? Have you compared the action distributions of the teleoperated and generated data?
  • App. A.1 says the agent sets mass, friction and CoM and checks them with interaction tests. Were these values compared against measurements, and is there any downstream comparison using the plain GS-Mesh scene?

Comments on the paper

3 comments
Reviewer_813nOpenReview ↗

General Assessment: Support for Acceptance and Oral/Spotlight Consideration

After reviewing the rebuttal and revised manuscript, I support acceptance with a rating of 4 and recommend consideration for an oral or spotlight presentation. The revision strengthens the evidence for the paper’s central contributions and makes its practical significance considerably clearer.

The paper addresses an important bottleneck in humanoid learning: obtaining scalable training data for tasks that require coordinated locomotion and manipulation. Its contribution is a complete framework connecting visually faithful reconstruction, interaction-ready physical scene construction, whole-body demonstration generation, and real-world policy deployment. The motion–environment co-adaptation mechanism addresses a concrete difficulty in this setting: a reference motion can remain balanced during execution while losing the contact relationships needed to complete the task. Using execution feedback to restore those relationships, followed by complete physical validation, provides a useful approach to generating executable interaction data.

The real-robot evaluation is a particular strength. The tasks include mobile pick-and-place, low-reaching manipulation, bimanual transport, and foot-assisted interaction while carrying an object. These settings meaningfully exercise whole-body coordination. Demonstrating that generated data can support deployment without collecting real-robot demonstrations of the target tasks gives the framework clear practical value. The reuse of a small simulation source pool further establishes a promising route to increasing training coverage while limiting additional human collection effort.

The revised manuscript provides substantive answers to the central concerns raised during discussion. The authors clarify physical revalidation after correction, quantify retained spatial coverage, evaluate downstream benefits at equal demonstration counts, introduce a controlled LEGS comparison, and test held-out spatial combinations while excluding post-correction overlap. The expanded cost accounting and clarified generation statistics also improve the transparency of the experimental conclusions. These additions make the relationship between the proposed mechanism and the observed benefits much easier to assess.

I particularly appreciate that the paper now combines substantial system capability with experiments that help explain why the system works. The correction ablations, coverage analysis, and compositional evaluation offer useful insights for researchers working on humanoid data generation and sim-to-real learning. Its stage-wise evaluation also provides a useful structure for examining failures in long-horizon whole-body tasks.

In my assessment, the importance of the problem, the completeness of the approach, the real-world whole-body demonstrations, and the strengthened experimental support make this a compelling contribution. I recommend acceptance and believe the work merits serious consideration for oral/spotlight recognition.

We sincerely appreciate your encouraging assessment and recognition of the work’s practical significance.

Your support means a great deal to us, and we are grateful for the time and care you have devoted to evaluating the paper.

AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We thank all reviewers for their constructive comments and suggestions. The reviewers recognized our strong real-world whole-body loco-manipulation results (VFth, 1qVk, 813n), the practical value of our motion–environment co-adaptation mechanism (VFth, 813n), the promising scalability of generated data (1qVk, 813n), and the benefits of Gaussian rendering (VFth). During the rebuttal, we addressed the concerns below with clarifications and additional experiments. These updates are included in the revised manuscript (highlighted in blue).

Key ConcernNew Evidence & Revision Summary
1. Correction & Downstream Benefits (VFth, 1qVk, 813n)We clarified full physical re-execution from t = 0 after correction and quantified spatial-bin coverage retention (77.4%–100%). With 100 demonstrations per variant, the full method achieves 7/20 Task A real-world success vs. 4/20–5/20 for component ablations. These results support the benefits of execution-conditioned correction in both generation yield and downstream real-world performance.
2. Generalization (VFth, 1qVk, 813n)We conducted an equal-count BASE–EXPANDED comparison and a separate held-out combination evaluation, excluding the held-out combinations even after correction. The results show that expanding generated training coverage improves real-world success and that our policy generalizes to robot–object combinations excluded from training.
3. Comparison with LEGS (VFth, 1qVk)We added a controlled comparison against LEGS using the same source demonstrations, controller, scene, and renderer. Zero-HLM achieves 7/20 and 10/20 real-world task success on Tasks A and B, respectively, vs. 2/20 on each task for LEGS, alongside 2.3×–6.9× faster generation. It also supports the leg-critical interaction in Task C, which LEGS’s native primitives do not support.
4. Cost & Reporting (1qVk, 813n; co-training: VFth)Including reconstruction, modeling, calibration, and source collection, generating 2,400 demonstrations takes 27.14 h with 53 min (0.883 person-h) of human effort (220× lower than the extrapolated teleoperation effort; App. F). We also clarified generation attempt counts (App. B.4) and added action-coverage analysis for co-training (App. E).

After these additional experiments and clarifications, we are grateful that VFth and 813n raised their ratings in support of acceptance, with 813n also recommending oral/spotlight consideration. We believe that our additional evidence and revisions address the original concerns raised by 1qVk; while their follow-up assessment is still pending, we look forward to any further feedback. We sincerely appreciate the Area Chair’s time and consideration of our revised manuscript and the discussion above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.