acceptodds
Under review as a conference paper at ICLR 2027

HUMANO: A Large-Scale Benchmark for Generalist Humanoid Loco-Manipulation

Abstract

Recent advances in vision-language-action models and whole-body control have rapidly expanded the capabilities of humanoid loco-manipulation, enabling generalist policies to perform increasingly diverse tasks with a single model. However, existing benchmarks often focus on individual tasks or small task suites, making it difficult to systematically evaluate multi-task capabilities and generalization. To fill this gap, we present HUMANO, a large-scale benchmark for generalist humanoid loco-manipulation. HUMANO contains 100 tasks organized into eight complementary suites, spanning 90 interactive scenes and over 2K object assets, with 5.5K in-distribution whole-body demonstrations. The benchmark covers fundamental skills, goal-directed loco-manipulation, generalization across spatial configurations, objects, robot configurations, and goals, as well as long-horizon skill composition and state-dependent reasoning. HUMANO further provides seven controlled OOD settings covering visual, linguistic, and robot-initialization shifts, together with 27.5K OOD demonstrations for future training and adaptation studies. All policies operate through a shared SONIC-based motion-latent interface, enabling standardized comparison under a common whole-body execution framework. Experiments with representative imitation and vision-language-action policies demonstrate the emerging potential of generalist models for multi-task humanoid loco-manipulation. Although multi-task learning remains more challenging than single-task training, performance continues to improve with increasing demonstration scale. Preliminary sim-to-real results further show that policies trained on HUMANO simulation data can transfer executable whole-body behaviors to a physical humanoid robot.

Primary area:
datasets and benchmarks
Keywords:
Humanoid Manipulation, Loco-Manipulation, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fRmiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents one of the largest public benchmarks and datasets of its kind for simulated humanoid loco-manipulation. More than doubling the total available demonstration data compared to previous benchmarks. The benchmark contains 100 tasks across 8 suites (i.e. categories), situated in 90 simulated scenes. 5.5k demonstrations are provided for the various tasks, with additional 27.5k demonstrations for OOD settings. Low-level control is provided through a pre-trained SONIC low-level tracking controller. The framework provides build in support for domain randomization, such as randomizing lighting, camera view point, and sensor noise. The tasks span a wide range of loco-manipulation capabilities, ranging from short horizon tasks, such as grasping, to longer horizon tasks that require the agent to sequence multiple skills.

The systematic task and evaluation design provides a good framework for evaluating future humanoid loco-manipulation techniques. Overall, I think this work presents a valuable resource for the community, and will help accelerate future research on humanoid loco-manipulation.

Critical strengths

This work presents one of the largest datasets and benchmark of its kind of simulated humanoid loco-manipulation.

The task design provides a broad coverage of diverse loco-manipulation behaviors.

Systematic design for evaluating generation.

Critical weaknesses

While the experiments evaluated performance of various models trained on the proposed dataset, it does not evaluate the performance of models trained on the Humano compared to models trained on previous datasets, such as SIMPLE or HumanoidMimicGen.

The OOD settings evaluate generalization to variations of tasks within each suite and various environment randomizations, but they do not evaluate generalization to new held out tasks.

Questions for authors

Since Humano is providing substantially more demonstration data compared to previous benchmarks, it would be insightful to compare models trained on previous datasets and models trained on the Humano dataset on similar tasks. This will help to highlight any performance improvements that may result from the dataset provided by Humano. Do the authors observe any improvements to performance and generalization from training with the HUMANO dataset, compared to training with prior datasets?

The OOD settings mainly evaluate generalization to variations of similar tasks within each suite, but there are no held out tasks that evaluates generalization to new types of tasks. Have the authors tested the HUMANO models on new held-out tasks that are substantially different from those in the training set? How well can the model generalize to such OOD tasks?

3 comments

1. W1 & Q1: While the experiments evaluated performance of various models trained on the proposed dataset, it does not evaluate the performance of models trained on the Humano compared to models trained on previous datasets, such as SIMPLE or HumanoidMimicGen. Since Humano is providing substantially more demonstration data compared to previous benchmarks, it would be insightful to compare models trained on previous datasets and models trained on the Humano dataset on similar tasks. This will help to highlight any performance improvements that may result from the dataset provided by Humano. Do the authors observe any improvements to performance and generalization from training with the HUMANO dataset, compared to training with prior datasets?

Response: We thank the reviewer for this insightful suggestion. In our original experiments, Diffusion Policy, ACT, and Flow Matching were trained from scratch using HUMANO demonstrations, whereas pretrained foundation models (e.g., π0.5\pi_{0.5}, GR00T N1.7, and Ψ0\Psi_0) were adapted using the corresponding in-distribution demonstrations and evaluated under both ID and OOD settings. These experiments primarily assess policy performance on HUMANO, rather than explicitly comparing the downstream benefits of different pre-training datasets.

To address this concern, we introduce an additional pre-training comparison using GR00T N1.7 as the common model initialization. Specifically, we continue pre-training the model for 10 epochs using either the training data from all eight HUMANO suites or the existing SIMPLE dataset [1]. Both models are subsequently post-trained for five epochs on the same HUMANO Basic suite and evaluated on Basic ID tasks and HUMANO-Plus OOD settings. We additionally include a no-pretraining baseline trained on Basic for 73 epochs. The results are as follows:

Table: Effect of Pretraining Data on Downstream Performance. We report in-distribution (ID) success rates on HUMANO Basic and out-of-distribution (OOD) success rates on HUMANO-Plus (%). Ep. denotes training epochs. Bold denotes the best result.

PretrainPre. Ep.Basic Ep.IDDistr.Light.Backg.CameraLang.SensorInit.OOD Avg.
None–7321.617.216.024.88.822.09.212.015.7
HUMANO10554.435.648.862.415.254.429.643.641.4
SIMPLE10516.47.612.414.80.820.03.27.69.5

It achieves 54.4% ID success and 41.4% OOD average success, substantially outperforming SIMPLE pre-training (16.4% and 9.5%, respectively). Notably, HUMANO pre-training also substantially outperforms the no-pretraining baseline (21.6% ID and 15.7% OOD), despite requiring only five epochs of downstream post-training compared with 73 epochs of Basic training. Furthermore, Figure 5(a) shows that the performance advantage of HUMANO pre-training persists throughout downstream fine-tuning, consistently outperforming SIMPLE pre-training across all evaluated checkpoints. These results provide direct empirical evidence that pre-training on HUMANO offers substantial benefits for downstream humanoid manipulation performance and robustness to distribution shifts, compared with pre-training on the existing SIMPLE dataset.

Revision: We have added Table 4 and Figure 5(a) to the revised manuscript, and have also added the corresponding analysis in Section 4.4.

2. W2 & Q2: The OOD settings evaluate generalization to variations of tasks within each suite and various environment randomizations, but they do not evaluate generalization to new held out tasks. The OOD settings mainly evaluate generalization to variations of similar tasks within each suite, but there are no held out tasks that evaluates generalization to new types of tasks. Have the authors tested the HUMANO models on new held-out tasks that are substantially different from those in the training set? How well can the model generalize to such OOD tasks?

Response: We thank the reviewer for this valuable suggestion. To further investigate generalization to unseen tasks beyond the seven controlled OOD settings, we evaluate GR00T N1.7 on three additional held-out tasks that are not explicitly included in the HUMANO training set. These tasks introduce different levels of novelty in object identity, manipulation behavior, and required skills. We conduct 25 evaluation rollouts per task, without using task-specific training demonstrations.

Held-out TaskTask DescriptionSuccess Rate (%)
Pear PickingPick up a pear from a table8.0 (2/25)
Drawer ClosingClose an open drawer84.0 (21/25)
Bottle ReorientationGrasp a horizontally placed bottle, reorient it upright, and place it stably on a table0.0 (0/25)

The results reveal substantial differences in generalization across unseen tasks. GR00T achieves 8.0% success on Pear Picking, suggesting partial transfer of grasping skills learned from similar objects, such as apples. Interestingly, it achieves 84.0% on Drawer Closing despite being trained on drawer-opening tasks, indicating that previously learned articulated-object interactions may transfer to related behaviors. In contrast, Bottle Reorientation, which requires a substantially different manipulation sequence involving grasping, in-hand or arm-mediated reorientation, and stable placement, achieves 0% success. These findings indicate that current VLA policies exhibit a certain degree of generalization to unseen tasks, but generalizing to substantially novel manipulation skills remains challenging.

Revision: We have added these OOD experiments and the corresponding analysis in Appendix E.3.

[1] SIMPLE: Simulation-Based Policy Learning and Evaluation for Humanoid Loco-manipulation. CoRL 2026.

Dear Reviewer fRmi,

With the rebuttal deadline approaching, we would greatly appreciate any further feedback on our responses. Please let us know if any concerns remain, and we would be happy to address them.

Thank you for your time!

Reviewer_5trxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HUMANO, a simulation-based benchmark for generalist humanoid loco-manipulation, with 100 tasks across eight capability suites, 90 scenes, over 2000 object assets, 5590 in-distribution demonstrations, and seven controlled OOD settings. It standardizes policy execution through a shared SONIC-based motion-latent interface and evaluates six imitation-learning and vision-language-action baselines. The results show substantial variation across tasks and suites: GR00T N1.7 achieves the highest task-weighted average success rate among the evaluated methods, while performance remains low on several goal-conditioned and conditional tasks. The paper also studies demonstration scaling and single-task versus multi-task learning, and reports preliminary sim-to-real transfer on a chair-pushing task.

Critical strengths
  1. Ambitious scope and structured task suite design. The benchmark attempts to move beyond small-scale humanoid evaluations by proposing 100 tasks across eight distinct capability suites, supported by 90 interactive scenes and over 2,000 object assets. The structured categorization (e.g., isolating spatial, object, and goal generalization, alongside long-horizon composition and conditional reasoning) provides a principled blueprint for analyzing distinct failure modes.

  2. Factorized OOD evaluation protocol. Introducing seven controlled evaluation axes provides a standardized framework for diagnostic robustness testing, moving past ad-hoc environment variations.

  3. Large-scale demonstration dataset. The benchmark provides 5.5K in-distribution whole-body demonstrations (23.2 hours) and releases 27.5K OOD trajectories, accompanied by an egocentric stereo-rendering pipeline separating interactive collection in MuJoCo from visual rendering in Isaac Sim.

Critical weaknesses
  1. Overly restrictive coupling to a specific hierarchical assumption and proprietary controller (SONIC). (Major Concern) The benchmark’s primary methodological limitation is that it strictly enforces a two-level hierarchical paradigm and hard-codes a single, frozen whole-body controller—SONIC. Whether high-level generalist policies should predict compact motion latents rather than end-to-end joint position/torque commands, Cartesian impedance targets, or alternative low-level representations is an open, actively debated question in humanoid robotics. By requiring all policies to predict a 64-D SONIC motion token and two 7-D hand commands, the benchmark cannot evaluate native, end-to-end humanoid policies without fundamentally retraining them on an external action interface. This design choice narrows HUMANO from a general humanoid loco-manipulation benchmark into an evaluation suite for SONIC-conditioned high-level planners.

  2. Under-specified sim-to-real validation. The physical validation is confined to a single task (Push Chair Forward, lines 467–477). The paper reports a success rate of 66.7% for GR00T N1.7 without reporting the number of trials, error distributions, or the failure modes encountered on the real G1 hardware. This provides only anecdotal evidence of real-world transferability and falls short of validating the simulation benchmark's fidelity across complex dynamic loco-manipulation.

Questions for authors
  1. How can policies with native joint-level or alternative action representations be fairly evaluated without retraining on SONIC?
  2. How do you disentangle high-level policy failures from low-level controller limitations?
  3. Why are the Long-Horizon and Movement suites omitted from the main evaluation, and will complete 100-task results be provided?
  4. What are the sample size and experimental details behind the sim-to-real transfer result?
4 comments

1. W1: Overly restrictive coupling to a specific hierarchical assumption and proprietary controller (SONIC). (Major Concern) The benchmark’s primary methodological limitation is that it strictly enforces a two-level hierarchical paradigm and hard-codes a single, frozen whole-body controller—SONIC. Whether high-level generalist policies should predict compact motion latents rather than end-to-end joint position/torque commands, Cartesian impedance targets, or alternative low-level representations is an open, actively debated question in humanoid robotics. By requiring all policies to predict a 64-D SONIC motion token and two 7-D hand commands, the benchmark cannot evaluate native, end-to-end humanoid policies without fundamentally retraining them on an external action interface. This design choice narrows HUMANO from a general humanoid loco-manipulation benchmark into an evaluation suite for SONIC-conditioned high-level planners.

Response: We thank the reviewer for raising this important concern.

(1)We respectfully disagree that HUMANO is merely an evaluation suite for SONIC-conditioned high-level planners. This concern reflects different interpretations of "general." Rather than comparing complete humanoid robotic systems, HUMANO aims to fairly evaluate the generalist capabilities of diverse policies and humanoid foundation models across a sufficiently broad range of tasks and evaluation settings, encompassing 8 complementary suites, 7 OOD settings, and multi-task training and evaluation, thereby addressing an important gap in existing benchmarks.

(2)Does HUMANO enforce a specific hierarchical paradigm and WBC? This characterization is not entirely accurate. Existing humanoid policies can be broadly categorized into two paradigms: (i) hierarchical architectures comprising a high-level policy and a low-level WBC, which are currently widely adopted (e.g., WholeBodyVLA [1], Ψ0\Psi_0[2], and HEX[3]); and (ii) end-to-end architectures that directly predict joint targets without a separate WBC (e.g., DPC [4]), only one method and not open-sourced. Within the hierarchical paradigm, high-level policies can predict either control commands for command-based WBCs (e.g., HOMIE and AMO) or motion latents for motion-tracking WBCs (e.g., SONIC).

In fact, HUMANO already supports alternative WBCs, including AMO and TWIST2, along with their corresponding data collection pipelines. However, recollecting demonstrations and retraining baselines across all 100 tasks for each WBC would require substantial human and hardware resources. Such comparisons would primarily investigate policy–WBC interactions, which are beyond our focus on benchmarking foundation model capabilities. We will clarify these supported interfaces in the revised manuscript.

(3) Why SONIC? Our choice of SONIC is driven by the diversity of HUMANO's 100 tasks. Unlike command-based controllers such as HOMIE and AMO, which expose predefined locomotion and posture commands, SONIC offers a unified motion representation that supports diverse whole-body behaviors (e.g., walking, crouching, sitting, and coordinated manipulation) within a single controller, enabling broad atomic skill coverage and long-horizon compositions without task-specific control primitives or additional controllers.

(4) Adaptation of pretrained baselines. The pretrained π0.5\pi_{0.5} predicts actions for wheeled robots, whereas Ψ0\Psi_0 predicts humanoid robot actions and control commands. We acknowledge that adapting these models to HUMANO requires modifying their action experts to predict the 64-D SONIC motion token and two 7-D hand commands. However, adapting the action representation during downstream fine-tuning is a widely adopted practice in humanoid foundation models, as exemplified by Ψ0\Psi_0 and MotionWAM [5]. Although the action representations used in pretraining and fine-tuning may differ, we believe that such evaluations still provide meaningful initial insights into the capabilities of these models. In contrast, this concern does not apply to our from-scratch baselines (Diffusion Policy, ACT, and Flow Matching), which are trained directly on HUMANO's action representation.

Revision: Section 3.4 elaborates on the second point.

2. W2: Under-specified sim-to-real validation. The physical validation is confined to a single task (Push Chair Forward, lines 467–477). The paper reports a success rate of 66.7% for GR00T N1.7 without reporting the number of trials, error distributions, or the failure modes encountered on the real G1 hardware. This provides only anecdotal evidence of real-world transferability and falls short of validating the simulation benchmark's fidelity across complex dynamic loco-manipulation. Q4: What are the sample size and experimental details behind the sim-to-real transfer result?

Response: Thank you for highlighting the limitations of our initial real-world validation. We have expanded our sim-to-real evaluation. For sim-to-real evaluation, we use a Unitree G1 humanoid robot equipped with BrainCo Revo2 dexterous hands and consider four representative tasks. We collect 40 simulation demonstrations per task, resulting in 160 demonstrations in total. Three representative policies (π0.5\pi_{0.5}, Ψ0\Psi_0, and GR00T N1.7) are independently trained for each task using only the corresponding single-task demonstrations, following their default implementation and training configurations described in the paper. We evaluate each policy on the same tasks in both simulation and the real world, conducting 20 simulation rollouts and 10 real-world trials per task to examine the consistency between simulation and real-world performance.

(1) Broader sim2real evaluation.

Taskπ₀.₅ Simπ₀.₅ RealΨ₀ SimΨ₀ RealGR00T SimGR00T Real
Apple Picking752075208030
Chair Pushing755085608580
Banana into Basket553060407050
Basket Moving451050105520
Average62.527.567.532.572.545.0

(2) Failure modes. Apple Picking: Failures are primarily caused by sim-to-real geometric discrepancies, which lead to inaccurate object localization and grasp positioning. Chair Pushing: The robot occasionally terminates its pushing motion prematurely, failing to push the chair fully underneath the table. Banana into Basket: Failures mainly arise from inaccurate banana grasping. In some cases, the robot accidentally collides with the tray while grasping the banana, causing the tray to be displaced. Basket Moving: The main failure modes include inaccurate grasping and overshooting the target position during the subsequent turning-and-walking motion.

Revision: We revise the main text by adding Table 6, Figure 7, and the corresponding analysis of the sim-to-real evaluation across four representative tasks. We also provide sim-to-real videos in the supplementary material.

3. Q1: How can policies with native joint-level or alternative action representations be fairly evaluated without retraining on SONIC?

Response: We believe that ensuring fairness across heterogeneous action representations remains an open challenge in humanoid foundation model evaluation

(1)In comparisons of humanoid foundation models, consider π0.5\pi_{0.5} and Ψ0\Psi_0 as examples. The pretrained π0.5\pi_{0.5} predicts actions for wheeled robots, whereas Ψ0\Psi_0 predicts humanoid robot actions and control commands. However, existing evaluations commonly overlook the mismatch between pretraining and post-training action representations, which may inherently disadvantage models pretrained on non-humanoid data. Nevertheless, results reported in HEX, ω\omega-0 [6], and λ0\lambda_0 [7] show that these two models achieve comparable performance, with Ψ0\Psi_0 even underperforming in some cases. We therefore believe that adaptation using the same downstream data and training protocol provides a reasonable and practically meaningful comparison, although it cannot completely eliminate pretraining-related differences.

(2)As discussed in W1(4), adapting action representations during downstream fine-tuning is widely adopted in humanoid foundation models. HUMANO follows this established practice to compare heterogeneous pretrained policies under a standardized action interface.

4. Q2: How do you disentangle high-level policy failures from low-level controller limitations?

Response: The sources of failure differ between simulation and real-world evaluations.

(1) Simulation failures. In simulation, we attribute task failures primarily to the high-level policy. All 100 HUMANO tasks have been successfully demonstrated using the SONIC-based teleoperation system, confirming that the WBC can execute the required motions for these tasks. Therefore, failures during policy evaluation mainly arise from inaccurate high-level predictions that fail to reproduce the demonstrated behaviors, rather than from fundamental WBC limitations.

(2) Real-world failures. On physical robots, failure attribution is more complex, as multiple factors can contribute. These include inaccurate high-level policy predictions, limitations in low-level control precision, and hardware-related issues such as actuator heating, which may cause motion drift even under identical action commands. Consequently, real-world failures may result from individual factors or their interactions, making precise attribution more challenging. Humanoid robotics still has a long way to go.

5. Q3: Why are the Long-Horizon and Movement suites omitted from the main evaluation, and will complete 100-task results be provided?

Response: Thank you for highlighting this omission. We have updated Table 2 to cover all 100 tasks across eight suites, including the previously omitted Movement (10 tasks) and Long (20 tasks) suites. The complete evaluation comprises 15,000 rollouts across six baselines, with the newly added results summarized below.

MethodMovementLong
Diffusion Policy11.60.6
Flow Matching20.03.2
ACT29.61.2
π0.5\pi_{0.5}38.43.8
GR00T N1.736.414.6
Ψ0\Psi_038.41.0

Revision: We have updated Table 2 and the corresponding analysis in the revised manuscript accordingly.

Additionally, we have expanded the OOD evaluation from the Basic suite to all eight suites, conducting 105K rollouts (100 tasks × 7 evaluation settings × 6 baselines × 25 rollouts). We have updated Table 3 and the corresponding analysis, with detailed suite-wise OOD results provided in the Appendix D.2.

Reference

[1] WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control. ICLR 2026.

[2] Ψ0: An Open Foundation Model Towards Universal Humanoid Loco-Manipulation. RSS 2026.

[3] HEX: Humanoid-Aligned Experts for Cross-Embodiment Whole-Body Manipulation. RSS workshop 2026.

[4] Direct Perception Control Model. Symbiosis Robotics Blog 2026.

[5] MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulation. arXiv 2026.

[6] ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation. arXiv 2026.

[7] Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining. arXiv 2026.

Dear Reviewer 5trx,

With the rebuttal deadline approaching, we would greatly appreciate any further feedback on our responses. Please let us know if any concerns remain, and we would be happy to address them.

Thank you for your time!

Reviewer_M4EBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HUMANO introduces a simulation benchmark with 100 humanoid loco-manipulation tasks across eight capability suites, 90 indoor scenes, and over 2,000 object assets. It provides 5.5K in-distribution and 27.5K OOD demonstrations, a shared SONIC-based controller, and evaluations of six policies on 70 tasks. The paper additionally studies seven OOD conditions and presents preliminary sim-to-real transfer.

Critical strengths

The benchmark provides a structured and extensive task inventory. Its eight suites cover basic manipulation, spatial and object variation, goal conditioning, temporal composition, and conditional decision-making. The 90 indoor scenes and 2K+ assets create a useful foundation for evaluating diverse humanoid behaviors (Section 3).

The standardized execution interface enables meaningful baseline comparisons. All six policies use the same SONIC-based whole-body controller and shared evaluation configurations. The task-level results reveal different capability profiles: π0.5 achieves 96% success on BallKick but 8% on ChairPush, whereas GR00T N1.7 achieves 24% and 100%, respectively (Table 12). The controlled OOD settings further support reproducible robustness comparisons.

Critical weaknesses

The experiments do not establish the claimed generalist benchmark coverage. Although the introduction states that all 100 tasks are benchmarked, Table 2 evaluates only 70, excluding the Movement and Long-Horizon suites.

The benchmark's validity as a measure of real-world capability remains insufficiently established. Comparing six policies demonstrates performance differences in simulation, but does not establish whether those differences predict physical-robot performance. If authors can justify / clear / explain it, I am happy to discuss potential better score.

Questions for authors

Can the authors evaluate several policy–task pairs in both simulation and reality to measure ranking agreement? For tasks with near-zero success, can intermediate progress or failure metrics provide more discriminative evaluation?

3 comments

1. W1: The experiments do not establish the claimed generalist benchmark coverage. Although the introduction states that all 100 tasks are benchmarked, Table 2 evaluates only 70, excluding the Movement and Long-Horizon suites.

Response: Thank you for highlighting this omission. We have extended Table 2 to cover all 100 tasks across 8 suites, including Movement (10 tasks) and Long (20 tasks), completing 15K rollouts across six baselines. The additional results are shown below.

MethodMovementLong
Diffusion Policy11.60.6
Flow Matching20.03.2
ACT29.61.2
π0.5\pi_{0.5}38.43.8
GR00T N1.736.414.6
Ψ0\Psi_038.41.0

Furthermore, we have expanded OOD evaluation from Basic to all 8 suites, totaling 105K rollouts (100 tasks × 7 settings × 6 baselines × 25 rollouts).

Revision: Tables 2–3 and their analyses have been updated accordingly, with detailed suite-wise OOD results provided in the Appendix D.2.

2. W2: The benchmark's validity as a measure of real-world capability remains insufficiently established. Comparing six policies demonstrates performance differences in simulation, but does not establish whether those differences predict physical-robot performance. If authors can justify / clear / explain it, I am happy to discuss potential better score. Q1: Can the authors evaluate several policy–task pairs in both simulation and reality to measure ranking agreement? For tasks with near-zero success, can intermediate progress or failure metrics provide more discriminative evaluation?

Response: We thank the reviewer for this valuable suggestion. For sim-to-real evaluation, we use a Unitree G1 humanoid robot equipped with BrainCo Revo2 dexterous hands and consider four representative tasks. We collect 40 simulation demonstrations per task, resulting in 160 demonstrations in total. Three representative policies (π0.5\pi_{0.5}, Ψ0\Psi_0, and GR00T N1.7) are independently trained for each task using only the corresponding single-task demonstrations, following their default implementation and training configurations described in the paper. We evaluate each policy on the same tasks in both simulation and the real world, conducting 20 simulation rollouts and 10 real-world trials per task to examine the consistency between simulation and real-world performance.

Sim2real success rates:

Taskπ₀.₅ Simπ₀.₅ RealΨ₀ SimΨ₀ RealGR00T SimGR00T Real
Apple Picking752075208030
Chair Pushing755085608580
Banana into Basket553060407050
Basket Moving451050105520
Average62.527.567.532.572.545.0

Across the four evaluated tasks, GR00T N1.7 achieves the highest average success rate in both simulation (72.5%) and reality (45.0%), followed by Ψ0\Psi_0 (67.5% and 32.5%) and π0.5\pi_{0.5} (62.5% and 27.5%). At the full benchmark level, however, the aggregate ranking is GR00T N1.7 > π0.5\pi_{0.5} > Ψ0\Psi_0 on both HUMANO ID and HUMANO-Plus OOD evaluations. Thus, while GR00T consistently ranks first, the relative ordering of the other two policies varies across evaluation sets.

Real-world progress scores:

Taskπ₀.₅Ψ₀GR00T N1.7
Apple Picking0.4250.4500.550
Chair Pushing0.7250.8250.850
Banana into Basket0.5500.6750.700
Basket Moving0.4250.4000.500
Average0.5310.5880.650

The progress scores further reveal differences in partial task completion, particularly when policies achieve similar binary success rates. For example, on Basket Moving, π0.5\pi_{0.5} and Ψ0\Psi_0 both achieve 10% success rates, while their progress scores are 0.425 and 0.400, respectively. This provides a more fine-grained view of policy execution beyond binary task success.

Revision: We add Table 6 to report quantitative sim2real comparisons across four representative tasks and Figure 7 to visualize the corresponding execution results. We also provide sim2real videos in supplementary material.

Thank you for the detailed response and additional experiments. The expanded evaluation addresses my concern about task coverage, and the real-world experiments provide useful preliminary evidence of sim-to-real ranking consistency. However, the substantial gap in success rates between simulation and reality remains a concern. Could real-world rollouts be used to calibrate simulation parameters and reduce this gap? It would also help to clarify how the four real-world tasks were selected. Overall, the response has addressed my main concerns, and I am inclined to raise my score.

For future version, I encourage the authors to discuss recent humanoid loco-manipulation approaches such as “InterMimicGen: Scaling Humanoid Loco-Manipulation through Self-Evolving Motion Imitation,” “Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation,” and “GRAIL: Generating Humanoid Loco-Manipulation from World Models,” and clarify how HUMANO complements them in terms of data generation, task coverage, and sim-to-real evaluation.

We thank the reviewer for this valuable suggestion.

(1) We agree that real-world rollouts can help reduce the sim-to-real gap. We have already calibrated the cameras, adapted control commands for the BrainCo Revo2 hands, and matched object dimensions and masses using physical measurements. However, gaps remain, especially in contact-intensive tasks. For example, GR00T N1.7 achieves 85%/80% success in simulation/reality on Chair Pushing, but 80%/30% on Apple Picking, likely due to differences in contact dynamics, friction, and grasp stability. Further rollout-based calibration could help reduce these gaps. Importantly, all policies are trained only on simulation demonstrations, without real-world fine-tuning.

(2) We selected the four real-world tasks based on three considerations: consistency with HUMANO's simulation tasks, feasibility on the physical G1 given object availability and hardware constraints, and coverage of different capabilities. Together, these tasks cover grasping, whole-body pushing, pick-and-place, and object moving.

(3) Finally, we have added the missing references you pointed out to the Related Work section and clarified how HUMANO relates to these approaches.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

We sincerely thank the Area Chair and all reviewers for their constructive feedback. In response, we have made the following additions and revisions:

  1. Complete ID evaluation: Extended evaluation from 70 to all 100 tasks across eight HUMANO suites, adding the previously omitted Movement and Long-Horizon suites, with 15K rollouts across six baselines (Table 2).

  2. Expanded OOD evaluation: Extended the seven controlled OOD settings from the Basic suite to all eight suites, resulting in 105K evaluation rollouts across 100 tasks and six baselines (Table 3 and Appendix D.2).

  3. Expanded sim-to-real evaluation: Extended physical-robot experiments from one to four tasks using a Unitree G1 with BrainCo Revo2 dexterous hands. We evaluated three representative policies using 160 simulation demonstrations, 20 simulation rollouts, and 10 real-world trials per task (Table 6 and Figure 7). Sim-to-real execution videos are also included in the supplementary material.

  4. Pre-training data comparison: Added controlled comparisons of GR00T N1.7 pre-trained on HUMANO and SIMPLE, together with a no-pretraining baseline. HUMANO pre-training achieves 54.4% ID and 41.4% OOD success rates (Table 4 and Figure 5(a)).

  5. Unseen-task generalization: Added evaluations on three held-out tasks beyond the predefined OOD settings, including Pear Picking, Drawer Closing, and Bottle Reorientation (Appendix E.3).

  6. Alternative controller support: Clarified that HUMANO supports alternative whole-body controllers, including AMO and TWIST2, together with their corresponding data collection pipelines, and further explained the rationale for adopting SONIC as the unified motion interface (Section 3.4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.