- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work presents one of the largest public benchmarks and datasets of its kind for simulated humanoid loco-manipulation. More than doubling the total available demonstration data compared to previous benchmarks. The benchmark contains 100 tasks across 8 suites (i.e. categories), situated in 90 simulated scenes. 5.5k demonstrations are provided for the various tasks, with additional 27.5k demonstrations for OOD settings. Low-level control is provided through a pre-trained SONIC low-level tracking controller. The framework provides build in support for domain randomization, such as randomizing lighting, camera view point, and sensor noise. The tasks span a wide range of loco-manipulation capabilities, ranging from short horizon tasks, such as grasping, to longer horizon tasks that require the agent to sequence multiple skills.
The systematic task and evaluation design provides a good framework for evaluating future humanoid loco-manipulation techniques. Overall, I think this work presents a valuable resource for the community, and will help accelerate future research on humanoid loco-manipulation.
Critical strengths
This work presents one of the largest datasets and benchmark of its kind of simulated humanoid loco-manipulation.
The task design provides a broad coverage of diverse loco-manipulation behaviors.
Systematic design for evaluating generation.
Critical weaknesses
While the experiments evaluated performance of various models trained on the proposed dataset, it does not evaluate the performance of models trained on the Humano compared to models trained on previous datasets, such as SIMPLE or HumanoidMimicGen.
The OOD settings evaluate generalization to variations of tasks within each suite and various environment randomizations, but they do not evaluate generalization to new held out tasks.
Questions for authors
Since Humano is providing substantially more demonstration data compared to previous benchmarks, it would be insightful to compare models trained on previous datasets and models trained on the Humano dataset on similar tasks. This will help to highlight any performance improvements that may result from the dataset provided by Humano. Do the authors observe any improvements to performance and generalization from training with the HUMANO dataset, compared to training with prior datasets?
The OOD settings mainly evaluate generalization to variations of similar tasks within each suite, but there are no held out tasks that evaluates generalization to new types of tasks. Have the authors tested the HUMANO models on new held-out tasks that are substantially different from those in the training set? How well can the model generalize to such OOD tasks?
3 comments
2. W2 & Q2: The OOD settings evaluate generalization to variations of tasks within each suite and various environment randomizations, but they do not evaluate generalization to new held out tasks. The OOD settings mainly evaluate generalization to variations of similar tasks within each suite, but there are no held out tasks that evaluates generalization to new types of tasks. Have the authors tested the HUMANO models on new held-out tasks that are substantially different from those in the training set? How well can the model generalize to such OOD tasks?
Response: We thank the reviewer for this valuable suggestion. To further investigate generalization to unseen tasks beyond the seven controlled OOD settings, we evaluate GR00T N1.7 on three additional held-out tasks that are not explicitly included in the HUMANO training set. These tasks introduce different levels of novelty in object identity, manipulation behavior, and required skills. We conduct 25 evaluation rollouts per task, without using task-specific training demonstrations.
| Held-out Task | Task Description | Success Rate (%) |
|---|---|---|
| Pear Picking | Pick up a pear from a table | 8.0 (2/25) |
| Drawer Closing | Close an open drawer | 84.0 (21/25) |
| Bottle Reorientation | Grasp a horizontally placed bottle, reorient it upright, and place it stably on a table | 0.0 (0/25) |
The results reveal substantial differences in generalization across unseen tasks. GR00T achieves 8.0% success on Pear Picking, suggesting partial transfer of grasping skills learned from similar objects, such as apples. Interestingly, it achieves 84.0% on Drawer Closing despite being trained on drawer-opening tasks, indicating that previously learned articulated-object interactions may transfer to related behaviors. In contrast, Bottle Reorientation, which requires a substantially different manipulation sequence involving grasping, in-hand or arm-mediated reorientation, and stable placement, achieves 0% success. These findings indicate that current VLA policies exhibit a certain degree of generalization to unseen tasks, but generalizing to substantially novel manipulation skills remains challenging.
Revision: We have added these OOD experiments and the corresponding analysis in Appendix E.3.
[1] SIMPLE: Simulation-Based Policy Learning and Evaluation for Humanoid Loco-manipulation. CoRL 2026.
Dear Reviewer fRmi,
With the rebuttal deadline approaching, we would greatly appreciate any further feedback on our responses. Please let us know if any concerns remain, and we would be happy to address them.
Thank you for your time!
1. W1 & Q1: While the experiments evaluated performance of various models trained on the proposed dataset, it does not evaluate the performance of models trained on the Humano compared to models trained on previous datasets, such as SIMPLE or HumanoidMimicGen. Since Humano is providing substantially more demonstration data compared to previous benchmarks, it would be insightful to compare models trained on previous datasets and models trained on the Humano dataset on similar tasks. This will help to highlight any performance improvements that may result from the dataset provided by Humano. Do the authors observe any improvements to performance and generalization from training with the HUMANO dataset, compared to training with prior datasets?
Response: We thank the reviewer for this insightful suggestion. In our original experiments, Diffusion Policy, ACT, and Flow Matching were trained from scratch using HUMANO demonstrations, whereas pretrained foundation models (e.g., π0.5, GR00T N1.7, and Ψ0) were adapted using the corresponding in-distribution demonstrations and evaluated under both ID and OOD settings. These experiments primarily assess policy performance on HUMANO, rather than explicitly comparing the downstream benefits of different pre-training datasets.
To address this concern, we introduce an additional pre-training comparison using GR00T N1.7 as the common model initialization. Specifically, we continue pre-training the model for 10 epochs using either the training data from all eight HUMANO suites or the existing SIMPLE dataset [1]. Both models are subsequently post-trained for five epochs on the same HUMANO Basic suite and evaluated on Basic ID tasks and HUMANO-Plus OOD settings. We additionally include a no-pretraining baseline trained on Basic for 73 epochs. The results are as follows:
Table: Effect of Pretraining Data on Downstream Performance. We report in-distribution (ID) success rates on HUMANO Basic and out-of-distribution (OOD) success rates on HUMANO-Plus (%). Ep. denotes training epochs. Bold denotes the best result.
It achieves 54.4% ID success and 41.4% OOD average success, substantially outperforming SIMPLE pre-training (16.4% and 9.5%, respectively). Notably, HUMANO pre-training also substantially outperforms the no-pretraining baseline (21.6% ID and 15.7% OOD), despite requiring only five epochs of downstream post-training compared with 73 epochs of Basic training. Furthermore, Figure 5(a) shows that the performance advantage of HUMANO pre-training persists throughout downstream fine-tuning, consistently outperforming SIMPLE pre-training across all evaluated checkpoints. These results provide direct empirical evidence that pre-training on HUMANO offers substantial benefits for downstream humanoid manipulation performance and robustness to distribution shifts, compared with pre-training on the existing SIMPLE dataset.
Revision: We have added Table 4 and Figure 5(a) to the revised manuscript, and have also added the corresponding analysis in Section 4.4.