- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates how simulation-generated training data can reduce the reliance on real-world demonstrations for humanoid mobile manipulation. The proposed Zero-HLM framework combines Gaussian-based scene reconstruction, agent-assisted physical modeling, and whole-body motion generation from source demonstrations. It jointly adjusts reference motions and environment configurations based on execution feedback to reduce the impact of motion-tracking errors on interaction success. The generated data is used to train visuomotor policies. The authors evaluate the approach through real-world deployment across multiple mobile manipulation tasks, component ablations, and simulation benchmarks. The main contribution is a unified pipeline that integrates scene construction and whole-body interaction data generation, with demonstrated transfer to real robots. The central technical claim is that execution-aware joint adaptation of motions and environments can improve the effectiveness of training data generation.
Critical strengths
1. Provides meaningful evidence of real-world transfer.
The paper uses generated data to train mobile manipulation policies for tasks involving walking, low-level grasping, bimanual carrying, and foot contact. It demonstrates real-world deployment without using real demonstrations of the target tasks. The comparable performance to training with the same number of real demonstrations also supports the practical value of the generated data. This is the strongest evidence in favor of the paper.
2. Addresses practical interaction issues in whole-body motion tracking.
The authors recognize that kinematically valid reference motions do not always preserve the required hand–object relationships during execution. They propose corrections for different types of errors. The ablation results suggest that these corrections improve the data generation success rate within the proposed pipeline, showing practical engineering value.
Critical weaknesses
1. The independent contribution of execution-aware correction is not sufficiently verified.
The current ablations mainly report data generation success rates, while environment correction also changes object or target positions. Therefore, higher success rates may reflect a more effective generation process, but may also come with changes in the distribution of generated configurations. The paper does not sufficiently report target coverage after correction. It also lacks controlled baselines using the same source demonstrations, controller, and renderer to evaluate downstream benefits under equal coverage or generation budgets. In addition, it is unclear whether the full trajectory is physically revalidated after modifying the environment. These issues make it difficult to assess how much this mechanism improves the generation of useful training data for the target tasks.
2. The generalization experiments do not clearly separate expanded training coverage from performance on unseen configurations.
The main comparisons change both the amount of training data and the coverage of configurations. The appendix shows that the expanded positions and alternative object–target combinations are already included in the corresponding simulation training data. Thus, the results mainly support sim-to-real transfer after targeted simulation training. This is valuable, but does not establish generalization to configurations not covered during training. Task C, which best reflects leg-based interaction, is only evaluated under the BASE setting, and some target positions are coupled with the robot's final position. The lack of equal-data coverage comparisons and held-out configuration tests limits the evidence for the claimed generalization ability.
Questions for authors
A. Key Questions That May Affect My Evaluation
1. Method clarification: What is the full validation process after environment correction?
After applying Eqs. (3) and (4), is the entire trajectory re-executed from the modified initial scene? Please clarify the order of correction, replay, and acceptance checks. Can the final accepted trajectories be replayed successfully without further changes to the environment state? This directly affects my assessment of the physical validity of the generated data.
2. Key comparison: What benefits does the correction mechanism provide while maintaining target coverage?
Could the authors compare the full method with a demonstration transformation baseline that does not modify the environment, using the same source demonstrations, SONIC controller, and visual renderer? Please report the final configuration coverage and downstream real-world performance. The existing ablations have already collected successful trajectories for each variant, which could be used for this comparison. Showing that the full method reduces generation costs while maintaining coverage, or improves real-world performance under the same budget, would strengthen the contribution.
3. Generalization: How much do expanded coverage and compositional generalization contribute under equal data sizes?
Could the authors compare models trained on equal amounts of BASE and EXPANDED simulation data, evaluated on the same real-world configurations? In addition, could some "robot start position × manipulation target position" combinations be held out? Each factor should appear separately during training, but their combination should remain unseen. This would help evaluate compositional generalization and better clarify the scope of the generalization claims.
B. Minor Clarification: Statistical Reporting and Raw Counts in Data Generation Experiments
The following concerns are reporting details that I suggest checking. They are not reasons for rejection and do not affect my final score.
Table 5 on page 7 reports a 61% generation success rate for the full method on Task A under the BASE setting, and states that 100 successful trajectories were collected for each variant. However, Appendix B.4.1 and Figure 13 report a 33% generation success rate for Task A. These numbers may come from different configuration sets, experimental runs, or evaluation stages, but their relationship is not clearly explained. Please clarify the evaluation scope, number of successes, and total number of attempts for each result. Also, please specify whether retries after correction are counted as new attempts.
In addition, Figure 14 reports failure category percentages of 87% and 13% for Sim-Astra on Task C, while stating that the total number of attempts is unavailable. Please explain how these percentages were calculated and whether the corresponding category counts or summary records are available. It would also help to clarify how these statistics relate to the timing experiments in Table 2, including how failures and retries are counted in the reported time.
The current information is insufficient to conclude that these differences are numerical inconsistencies. However, there may be reporting errors, and I encourage the authors to check them. For now, these concerns are not reasons for rejection and do not affect my final score.
Discussion with Reviewer 1qVk: How the Revised Evidence Addresses the Main Concerns
Thank you for raising the questions about configuration correction, comparisons with existing methods, generalization, and the complete collection cost. These concerns overlap substantially with my initial reservations. After reviewing the authors’ responses and the revised manuscript, I found the additional evidence persuasive and have raised my rating to 4. I would like to highlight the changes that most affected my assessment.
Regarding whether moving objects simplifies the sampled tasks, Appendix B.4.4 now reports both correction magnitudes and the spatial coverage retained after correction. For targets requiring adjustment, median translations are 1.5–4.1 cm, while the reported robot-start and object-position coverage retention ranges from 77.4% to 100%. Together with complete physical re-execution and the new downstream ablations at equal demonstration counts, these results substantially alleviate my concern that the procedure succeeds primarily by concentrating the data into a narrow set of easier configurations. The revised evaluation now examines the consequences of correction for both coverage and policy learning.
The added LEGS comparison is another consequential improvement. The adapted baseline shares the source demonstrations, SONIC controller, reconstructed scene, and Gaussian renderer, while retaining its procedural motion-primitive generation approach. With 100 accepted demonstrations per task, Zero-HLM achieves full-task success of 7/20 and 10/20 on Tasks A and B, respectively, compared with 2/20 for LEGS on each task. It also requires less time to collect a fixed number of successful demonstrations. This directly addresses the absence of a specialized humanoid data-generation baseline and provides stronger evidence for the proposed approach beyond the shared controller and rendering infrastructure.
For generalization, the new held-out combination experiment is particularly relevant. Every individual robot-start and object position appears during training, but four combinations are excluded, including configurations after correction. The real-world results therefore provide evidence of compositional generalization beyond direct training coverage of the evaluated combinations. The evaluation remains within one reconstructed scene, and cross-scene generalization is appropriately left as future work. Within the paper’s stated scope, however, the evidence is materially stronger than in the original submission.
Appendix F also makes the cost discussion more informative by including reconstruction, configuration, calibration, and source-demonstration collection. The authors report 27.14 hours for the complete pipeline producing 2,400 successful demonstrations, with 53 minutes of active human involvement. They explicitly identify the large-scale real-teleoperation comparison as an extrapolation from smaller-scale measurements. This makes the practical benefit clearer: the initial preparation and source demonstrations can be reused to expand the dataset without repeatedly incurring human teleoperation effort.
Taken together, these additions substantially address the reasons that initially led me to question acceptance. The real-robot results, combined with the more informative controls and transparent accounting, now support a positive assessment in my view. I hope these findings are useful when reconsidering the revised submission, and I would welcome further discussion of any specific issue that remains consequential to your recommendation.