acceptodds
Under review as a conference paper at ICLR 2027

CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration

Abstract

Many physical tasks in human environments require collaboration, from assisting a partner to jointly manipulating an object. Yet, existing humanoid benchmarks largely focus on single-humanoid skills and lack evaluation of multi-humanoid collaboration under egocentric visual observations. We introduce CoHuB (Collaborative Multi-Humanoid Benchmark), a simulation benchmark for multi-humanoid collaboration under egocentric visual observations. CoHuB provides 10 tasks—eight with two humanoids and two with three humanoids—spanning diverse collaboration patterns. We also provide synchronized demonstrations collected through a multi-operator VR teleoperation pipeline, in which each operator controls one humanoid from its egocentric view. Experiments with representative visuomotor policies reveal substantial challenges across different forms of coordinated perception and control. CoHuB provides a foundation for developing and evaluating multi-humanoid collaboration policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Benchmark and Datasets, Humanoid Robots, Multi-Robot Collaboration, Simulation-based Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2yY5OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper contributes a new simulation benchmark for multi-agent humanoid teaming. 10 loco-manipulation tasks are provided that require collaboration between two or three Unitree humanoids, simulated in Isaac Lab. The tasks vary in terms of how much locomotion is required and how coupled the manipulation is (e.g. two robots hanging a picture frame). In addition to the simulation environment, the benchmark also includes demonstration data recorded by multiple human teleoperators who each control one robot through a VR headset. Lastly, it evaluates several popular and current locomanipulation policies. The evaluation includes some interesting findings; in particular, policies designed specifically for multi-agent learning are not the most performant on this benchmark.

Critical strengths
  • The paper is well-written and very easy to follow.

  • The benchmark appears to fill a current gap in robotics research; it includes some useful information about the current benchmark landscape and its task suite is well-justified.

  • The inclusion of demonstration data is a plus.

  • The evaluation of existing policy architectures is substantial, including multiple large foundation model-based policies

Critical weaknesses
  • The paper claims reproducibility, but the supplementary material does not appear to include any code, only videos. Also, the computational resources needed to reproduce the results may be prohibitive for many researchers.

  • The empirical success rate results do not have any indications of statistical significance, such as confidence intervals or standard deviations over multiple repetitions.

  • While benchmarks can be useful for the community, this paper does not contribute any new scientific knowledge (theoretical analyses, novel architectures, etc.). If ICLR is not interested in dataset papers, that could be an issue.

Questions for authors
  • Can the authors share the success rates of the VR demonstrations? That would be a useful reference point for interpreting the policy performance metrics.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2yY5 (1/2): W1, W2, W3

Thank you for the positive assessment and constructive feedback. Following your suggestions, we have added our code to the supplementary material and report results over three training seeds. Section, table and figure numbers below follow the revised paper, and "the original Table 2" refers to Table 2 of the initial submission (seed 0). Please find our response below.

  • "The paper claims reproducibility, but the supplementary material does not appear to include any code, only videos. Also, the computational resources needed to reproduce the results may be prohibitive for many researchers."

We have added our code to the supplementary material. We ran all training and evaluation on a mix of NVIDIA RTX 3090, RTX 4090, RTX A6000 and RTX PRO 6000 GPUs. For larger models such as Fast-WAM, we used high-memory GPUs such as the RTX PRO 6000 so that each training run fits on a single GPU.

  • "The empirical success rate results do not have any indications of statistical significance, such as confidence intervals or standard deviations over multiple repetitions."

Following your suggestion, we retrain ACT, DP, the three VLAs, and Fast-WAM with two additional seeds (three seeds in total) and evaluate them on the two-humanoid tasks. Due to computational costs and the limited rebuttal period, the three-seed results of LatentToM and GauDP and of the three-humanoid tasks will be included in the final version. The table below reports the task success rates (%) as mean ±\pm standard deviation over the three seeds, with 100 rollouts per task and seed. Its last column gives the average in the original Table 2 (seed 0), and the revised Table 2 includes the three-seed results.

HandoverPouringFrameHangCoPouringTrashCollectionCartServiceCoCarryTableAlignAvg.Original Avg.
ACT55.0$\pm$9.21.0$\pm$1.046.7$\pm$2.59.7$\pm$1.20.3$\pm$0.61.7$\pm$1.556.7$\pm$4.043.7$\pm$5.026.8$\pm$1.127.9
DP26.7$\pm$3.10.0$\pm$0.024.3$\pm$1.55.3$\pm$2.10.3$\pm$0.62.3$\pm$0.613.7$\pm$1.53.7$\pm$0.69.5$\pm$0.810.4
π0.5\pi_{0.5}2.3$\pm$0.60.0$\pm$0.023.7$\pm$4.90.3$\pm$0.60.0$\pm$0.03.3$\pm$1.21.3$\pm$0.61.3$\pm$0.64.0$\pm$0.84.5
GR00T N1.722.7$\pm$6.70.3$\pm$0.633.3$\pm$10.11.0$\pm$1.70.3$\pm$0.60.0$\pm$0.01.7$\pm$0.60.3$\pm$0.67.5$\pm$2.29.4
Ψ0\Psi_00.7$\pm$1.20.0$\pm$0.09.0$\pm$5.31.0$\pm$1.72.3$\pm$1.20.0$\pm$0.00.3$\pm$0.60.3$\pm$0.61.7$\pm$1.11.0
Fast-WAM4.7$\pm$3.111.7$\pm$6.428.0$\pm$1.016.3$\pm$13.81.3$\pm$1.51.0$\pm$1.024.3$\pm$9.113.3$\pm$4.712.6$\pm$1.512.1

The gaps between the average success rates in the original Table 2 hold across seeds. ACT remains clearly above DP, and Fast-WAM remains the best among the VLAs and the WAM.

  • "While benchmarks can be useful for the community, this paper does not contribute any new scientific knowledge (theoretical analyses, novel architectures, etc.). If ICLR is not interested in dataset papers, that could be an issue."

CoHuB is a benchmark contribution. ICLR has published many dataset and benchmark papers, including RoboCasa365 [1] and OGBench [2], as well as multi-agent embodied benchmarks such as Habitat 3.0 [3] and PARTNR [4]. Beyond the benchmark and dataset, our analyses focus on collaboration rather than on robustness to scenes, objects or instructions. We study where collaborative phases fail and how role information and parameter sharing affect multi-humanoid policies. These analyses explore how vision-based policies collaborate with each other, which remains largely underexplored.

References

[1] RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots. ICLR 2026.

[2] OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025.

[3] Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots. ICLR 2024.

[4] PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks. ICLR 2025.

We continue our rebuttal in the next comment.

AuthorsOpenReview ↗

Response to Reviewer 2yY5 (2/2): Q1

  • "Can the authors share the success rates of the VR demonstrations? That would be a useful reference point for interpreting the policy performance metrics."

The demonstrations were collected by expert operators who practiced each task before recording. We did not record exact success rates, but the operators estimate that they succeeded in over 90% of attempts on most tasks. The tasks with both high physical coupling and high base movement (CoCarry, TableAlign and BigTable) were harder, with estimated success rates of about 60-70%, because the operators must coordinate a shared load while walking without force feedback. Based on these operator estimates, the learned policies appear to remain below the operators overall, leaving substantial room for improvement in current policies.


Thank you again for your helpful comments. Please let us know if any question remains. If your concerns are addressed, we would be grateful if you would consider updating your score.

The rebuttal largely addresses my concerns, so I increased my rating. I don't have the computational resources to run the code, but since it has been attached to this openreview page which will be made public, the reproducibility issue is mitigated.

Dear Reviewer 2yY5,

We are glad that our rebuttal has largely addressed your concerns. Thank you again for your time and for raising your rating.

Thanks,

Authors

Reviewer_vfMjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CoHuB introduces a simulation benchmark for multi-humanoid collaboration, comprising ten tasks, synchronized demonstrations collected through multi-operator VR teleoperation, and evaluations of imitation-learning, VLA, and WAM policies.

Critical strengths

The benchmark addresses an important problem through diverse tasks involving handovers, sustained co-manipulation, transportation, and three-robot coordination.

The synchronized teleoperation pipeline and demonstration dataset provide useful infrastructure for future research.

Broad baseline coverage and phase-level evaluation offer a useful starting point for analyzing collaborative policy failures.

Critical weaknesses

Physical feasibility is insufficiently established. Table-carrying tasks with Dex3-1 hands raise concerns about whether the simulated loads are compatible with the hardware’s grasping and actuator capabilities. The paper does not adequately specify object masses, friction coefficients, hand torque limits, or per-hand loads. Without this information, it is difficult to assess whether successful policies exploit unrealistic actuation or contact settings. If furniture masses are substantially reduced, the implications for load sharing and physical coupling should be discussed.

Baseline comparisons involve substantial configuration differences. ACT and DP use task- and robot-specific policies, whereas VLA/WAM models share parameters across tasks and roles. Observation access, training budgets, and replanning frequencies also differ. Although these differences are documented, they limit conclusions about the relative capabilities of policy families.

Collaboration-specific difficulties are not sufficiently isolated. Lower success in collaborative phases may reflect differences in manipulation precision and phase complexity. The predominantly fixed roles and limited initialization variations also leave partner responsiveness underexplored. Partner delays, motion perturbations, and cross-policy pairing would provide more direct tests of adaptive coordination.

Some design conclusions require stronger controls. Shared-policy configurations change the availability of role information, complicating interpretation of parameter-sharing effects. Results from multiple training seeds would also strengthen confidence in the reported rankings.

Questions for authors

What are the furniture masses and measured per-hand forces and joint torques during transport? Are these consistent with realistic Dex3-1 limits?

Could baseline comparisons be supplemented with matched single-task or multi-task training and explicit role conditioning?

How do policies respond when a partner starts late, pauses, or follows a different policy?

5 comments
AuthorsOpenReview ↗

Response to Reviewer vfMj (1/3): W1/Q1

Thank you for the positive assessment and constructive feedback. Following your suggestions, we have reported the physical parameters and run additional experiments with multi-task training of ACT and DP, a late-starting or pausing partner, cross-policy pairing, an agent identifier and three training seeds. Section, table and figure numbers below follow the revised paper, and "the original Table 2" refers to Table 2 of the initial submission (seed 0). Please find our response below.

  • "Physical feasibility is insufficiently established. Table-carrying tasks with Dex3-1 hands raise concerns about whether the simulated loads are compatible with the hardware's grasping and actuator capabilities. The paper does not adequately specify object masses, friction coefficients, hand torque limits, or per-hand loads. Without this information, it is difficult to assess whether successful policies exploit unrealistic actuation or contact settings. If furniture masses are substantially reduced, the implications for load sharing and physical coupling should be discussed."

  • "What are the furniture masses and measured per-hand forces and joint torques during transport? Are these consistent with realistic Dex3-1 limits?"

In short, we chose the physical parameters to stay as close to the real G1 as possible while keeping the collaborative carrying tasks feasible in simulation. We summarize them below.

Object masses. Table 6 of the appendix now lists the mass of every task object (0.035-7.5 kg for the objects that the robots lift, and 12 kg for the service trolley, which is pushed on casters). We set the object masses so that each robot's share stays within the G1 EDU arm payload of about 3 kg per arm. Large objects such as the tables still need more than one robot, because they are too large for a single robot to hold level.

Hand settings and measured loads. The Unitree Dex3-1 URDF sets the finger torque limits to 1.4 N$\cdot$m (2.45 N$\cdot$m for the thumb rotation joint), and the Unitree product page lists ideal maximum torques of 0.76 N$\cdot$m and 1.498 N$\cdot$m for the two joint motor models of the hand. With the URDF limits, held objects slipped from the hand or could not be lifted at all. We therefore set the finger torque limits to the lowest values at which held objects no longer slipped, 8 N$\cdot$m (5 N$\cdot$m for the thumb rotation joint), and added finger pads with friction coefficients of 1.4 (static) and 1.2 (dynamic). We then measured the per-hand loads at the wrists in the demonstrations, and they exceed the rated payload of the real Dex3-1 hand in most tasks. We acknowledge this as a limitation of CoHuB. Nevertheless, the comparison between the baselines remains meaningful, because every method is trained and evaluated under the same physical settings. We have added the hand settings to Table 5 and this discussion to Appendix A.

We continue our rebuttal in the next comment.

AuthorsOpenReview ↗

Response to Reviewer vfMj (2/3): W2/Q2, W3/Q3

  • "Baseline comparisons involve substantial configuration differences. ACT and DP use task- and robot-specific policies, whereas VLA/WAM models share parameters across tasks and roles. Observation access, training budgets, and replanning frequencies also differ. Although these differences are documented, they limit conclusions about the relative capabilities of policy families."

  • "Could baseline comparisons be supplemented with matched single-task or multi-task training and explicit role conditioning?"

In the paper, we compare single-task and multi-task policies separately within their respective categories (Secs. 4.1-4.2 and Table 2). For each method, we followed the setting of its original paper as closely as possible, and LatentToM and GauDP receive additional partner information as in their original designs.

Following your suggestion, we additionally train ACT and DP in the same multi-task setting as the VLAs and the WAM, with the same batch size (32) and number of training steps (40k). A single policy is shared across all tasks and robot roles. Since ACT and DP take no language input, we condition them on one-hot encodings of the task (8 tasks) and the robot's role (2 roles), which give them the same task and role information that the VLAs and the WAM receive from the instructions. This allows a direct comparison with the pretrained models in the same setting. The table below reports the task success rates (%) for training seed 0.

TrainingHandoverPouringFrameHangCoPouringTrashCollectionCartServiceCoCarryTableAlignAvg.
ACTPer task (original Table 2)65144913613927.9
ACTMulti-task63045410017921.9
DPPer task (original Table 2)3002671312410.4
DPMulti-task006300001.1
π0.5\pi_{0.5}Multi-task (original Table 2)2027004124.5
GR00T N1.7Multi-task (original Table 2)27044010219.4
Ψ0\Psi_0Multi-task (original Table 2)005030001.0
Fast-WAMMulti-task (original Table 2)27281110311712.1

Multi-task training lowers the success of both policies (ACT 27.9% to 21.9%, DP 10.4% to 1.1%). In this matched setting, ACT still outperforms every VLA and the WAM on average, while DP falls below most of them.

  • "Collaboration-specific difficulties are not sufficiently isolated. Lower success in collaborative phases may reflect differences in manipulation precision and phase complexity. The predominantly fixed roles and limited initialization variations also leave partner responsiveness underexplored. Partner delays, motion perturbations, and cross-policy pairing would provide more direct tests of adaptive coordination."

  • "How do policies respond when a partner starts late, pauses, or follows a different policy?"

We agree that the gap between collaborative and non-collaborative phases in Fig. 4 may also reflect manipulation precision and phase difficulty. Following your suggestion, we evaluate how the policies respond when their partner starts late, pauses or follows a different policy, without retraining.

When Robot A starts 3 s late, the average success of GR00T N1.7 over the eight two-humanoid tasks drops from 9.4% to 0.4%, mainly in the tasks where both robots handle the same object from the start. A 3 s pause of Robot A from 2 s to 5 s also lowers it to 1.3%. The table below gives the phase-wise success rates (%), where p3p_3 is task success.

GR00T N1.7p1p_1p2p_2p3p_3
Original61.436.39.4
Robot A starts 3 s late32.013.90.4
Robot A pauses from 2 s to 5 s57.926.01.3

The delay and pause experiments show that GR00T N1.7 struggles to adjust its actions to changes in partner timing, providing further evidence that collaboration remains challenging. We have added the late-start and pause experiments to Sec. 4.4 and Appendix F.3.

We also evaluate ACT with DP and GR00T N1.7 with π0.5\pi_{0.5} without retraining. We use 100 rollouts per task. The table below gives the phase-wise success rates (%) averaged over the same eight tasks.

Robot ARobot Bp1p_1p2p_2p3p_3
ACTACT81.065.427.9
DPDP63.633.910.4
ACTDP75.841.114.0
DPACT67.140.613.9
GR00T N1.7GR00T N1.761.436.39.4
π0.5\pi_{0.5}π0.5\pi_{0.5}63.325.94.5
GR00T N1.7π0.5\pi_{0.5}58.933.07.8
π0.5\pi_{0.5}GR00T N1.763.632.66.6

The mixed teams complete some tasks without retraining, and their average task success lies between that of teams using either method alone.

We continue our rebuttal in the next comment.

AuthorsOpenReview ↗

Response to Reviewer vfMj (3/3): W4

  • "Some design conclusions require stronger controls. Shared-policy configurations change the availability of role information, complicating interpretation of parameter-sharing effects. Results from multiple training seeds would also strengthen confidence in the reported rankings."

In Table 4, the shared ACT and DP policies have no explicit role information. To isolate its effect, we train the ACT and DP Local+Shared policies again with a one-hot agent identifier as an additional input and keep everything else fixed. The table below compares the task success rates (%) without the identifier (Shared) and with it (Shared + ID), using 100 rollouts per task.

SettingHandoverPouringFrameHangCoPouringTrashCollectionCartServiceCoCarryTableAlignAvg.
ACTShared52147710414223.9
ACTShared + ID59050330584126.8
DPShared230303111359.5
DPShared + ID141322011999.8

The identifier improves the average success slightly (ACT 23.9% to 26.8%, DP 9.5% to 9.8%). The gain is largest on CoCarry (ACT 41% to 58%, DP 13% to 19%). In CoCarry, the two robots stand at opposite ends of the same basket, so their egocentric views alone may not be enough to tell the policy which robot it is controlling. This suggests that role ambiguity matters on such tasks. We have added this experiment to Sec. 4.5 and Appendix F.4.

As suggested, we also retrain ACT, DP, the three VLAs, and Fast-WAM with two additional seeds (three seeds in total) and evaluate them on the two-humanoid tasks. Due to computational costs and the limited rebuttal period, the three-seed results of LatentToM and GauDP and of the three-humanoid tasks will be included in the final version. The table below reports the task success rates (%) as mean ±\pm standard deviation over the three seeds, with 100 rollouts per task and seed. Its last column gives the average in the original Table 2 (seed 0), and the revised Table 2 includes the three-seed results.

HandoverPouringFrameHangCoPouringTrashCollectionCartServiceCoCarryTableAlignAvg.Original Avg.
ACT55.0$\pm$9.21.0$\pm$1.046.7$\pm$2.59.7$\pm$1.20.3$\pm$0.61.7$\pm$1.556.7$\pm$4.043.7$\pm$5.026.8$\pm$1.127.9
DP26.7$\pm$3.10.0$\pm$0.024.3$\pm$1.55.3$\pm$2.10.3$\pm$0.62.3$\pm$0.613.7$\pm$1.53.7$\pm$0.69.5$\pm$0.810.4
π0.5\pi_{0.5}2.3$\pm$0.60.0$\pm$0.023.7$\pm$4.90.3$\pm$0.60.0$\pm$0.03.3$\pm$1.21.3$\pm$0.61.3$\pm$0.64.0$\pm$0.84.5
GR00T N1.722.7$\pm$6.70.3$\pm$0.633.3$\pm$10.11.0$\pm$1.70.3$\pm$0.60.0$\pm$0.01.7$\pm$0.60.3$\pm$0.67.5$\pm$2.29.4
Ψ0\Psi_00.7$\pm$1.20.0$\pm$0.09.0$\pm$5.31.0$\pm$1.72.3$\pm$1.20.0$\pm$0.00.3$\pm$0.60.3$\pm$0.61.7$\pm$1.11.0
Fast-WAM4.7$\pm$3.111.7$\pm$6.428.0$\pm$1.016.3$\pm$13.81.3$\pm$1.51.0$\pm$1.024.3$\pm$9.113.3$\pm$4.712.6$\pm$1.512.1

The gaps between the average success rates in the original Table 2 hold across seeds. ACT remains clearly above DP, and Fast-WAM remains the best among the VLAs and the WAM.


Thank you again for your constructive feedback. We believe the additional experiments and the clarified physical settings have strengthened the paper. Please let us know if any concern remains. If our responses address your concerns, we would be grateful if you would consider updating your score.

Reviewer_vFKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission introduces cohub, a simulation benchmark in isaac lab for collaboration between two or three unitree g1 humanoids with three-finger hands. There are 10 tasks. Eight use two robots and are split into four categories by how much the robots move (low or high base movement) and how much they must handle the same object at the same time (low or high physical coupling), with two tasks per category. The other two tasks use three robots. Each task is split into three sequential phases with state-based success criteria. By default each robot sees its own egocentric rgb image and proprioception (plus a role instruction for language-conditioned models), and outputs upper-body joint targets plus a base velocity and height command. A pretrained locomotion controller drives the legs. Demonstrations (60 per task, 600 in total) are collected with a multi-operator vr setup built on isaac teleop, where each operator controls one robot in a shared simulation. States are recorded online and images are rendered offline. Eight policies are evaluated: two standard imitation methods (act, dp), two multi-agent imitation methods (latenttom, gaudp), three vlas (pi0.5, groot n1.7, psi0) and one world action model (fast-wam), plus an exploratory zero-shot run of an llm agent. Main results use 100 rollouts per task and a single training seed. Success is low on most tasks, though the best methods reach 60 to 80 percent on a few (e.g. bigtable, handover, cocarry). Act has the highest two-robot average and ties fast-wam on the three-robot average, and fast-wam is the strongest among the pretrained models. A phase-level evaluation reports lower success on collaborative than on non-collaborative phases, and an ablation on observation scope and parameter sharing reports effects that differ by model.

Critical strengths
  1. The task suite and evaluation protocol are specified with unusual care, which makes the benchmark easy to audit and its failures easy to locate.

-- The two design axes, base movement and physical coupling, are motivated by existing taxonomies of interactive behaviour, and they produce a real spread: from a handover at a counter, to co-carrying a basket to a shelf or a table across an office, to two robots carrying a table through a doorway that a third robot opens. The ten tasks sit in ten different scenes, from kitchens and an art gallery to a laundromat, an office and a multi-room home, adapted from public scene assets.

-- Every task has three cumulative phases with explicit, physically meaningful thresholds: lift heights in centimeters, tilt limits, settling velocities, a minimum release duration, and alignment tolerances such as a table set square to a chair within 4 degrees. Because each phase requires the previous ones, a correct final pose alone does not count as success. The full criteria, randomization ranges, episode limits and language instructions are all listed in the appendix.

-- The protocol is controlled in a way many benchmarks are not: all trained policies are evaluated on the same environment seeds (1 to 100), with the same train/validation split. The phase-level evaluation, which restarts each phase from demonstration states, is a sensible way to look at later phases without earlier failures masking them.

-- The phase breakdown is put to good use. The qualitative analysis shows that two models with the same 2 percent handover success fail differently (one collides with the partner's arm during the transfer, the other completes the transfer but does not release), and that in cocarry and tablealign vlas tend to drop the object in transport while the world action model keeps its grasp but the two robots move inconsistently. The failure figure covers all ten tasks.

-- The paper is candid about its own caveats: the extra inputs some multi-agent baselines get, the llm agent not being directly comparable, and the possible confound between collaboration and phase difficulty. The reported numbers are internally consistent between the main table, the phase-wise tables and the text. The supplementary videos show reasonable physical realism on many tasks, for example the three-robot table carry with a fairly coherent gait.

  1. The data, interface and baseline infrastructure are built to be reused.

-- The teleoperation setup ties the demonstrations to what the policy will see. Each operator controls one robot from that robot's own egocentric view, and the headset shows the boundary of the onboard camera's field of view. A shared recording session keeps physics consistent across the operators' mirror sessions.

-- Recording only states and actions online, then replaying trajectories to render images, cuts the online overhead and in principle allows re-rendering with other sensors later. Besides egocentric rgb, the dataset keeps depth, third-person views, object poses, grasp status and wrist force-torque. Most of these are not used by the baselines, but they are clearly useful for future work.

-- All tasks share one per-robot interface (31 upper-body joint targets plus a 4-d base command, 43 joint positions, one head camera with stated resolution and field of view), following the decoupled whole-body control convention used for gr00t on the g1. Scene assets are kept separate from robot and task logic. Together these make it straightforward to add models or tasks.

-- The baseline set covers four policy families, including recent humanoid vlas, and the training recipes are reported in considerable detail (initialization, batch size, budget, learning rate, lora ranks, action chunking). Each task also comes with a joint instruction and one instruction per role, which supports language-conditioned work.

-- The exploratory llm-agent run is timely, and it gives a profile quite different from the learned policies: 70 percent on co-pouring, where no learned policy exceeds 11 percent, 10 percent on pouring, and no successes on the other two-robot tasks. Physical collaboration among several humanoids from egocentric vision has little shared infrastructure today, and this would be one.

Critical weaknesses

W1. The submission reads as a dataset and systems effort without a clear learning question, and its related work leaves out the visual multi-agent and cooperative marl work closest to it.

-- There is no formal problem statement. The paper does not define a dec-pomdp or similar, does not state a communication model (explicit inter-agent communication is only mentioned as future work), and does not state a sensing-noise model or localization assumptions. The baselines run under quite different regimes (fully decentralized, centralized training with decentralized execution plus a shared third-person view, fully centralized), so it is unclear which learning problem the benchmark is meant to pose. With no sim-to-real evidence, the case for the benchmark has to come from what it reveals about learning methods, and the analysis beyond success rates (phase-level results, qualitative failures, one ablation) does not yet isolate a cause on the learning or coordination side.

-- Sec 4.3 finds collaborative phases harder, but the authors note this may reflect phase difficulty. Sec 4.4 concludes that the effects of observation scope and sharing vary across models. Neither result is followed by an experiment that isolates the cause, e.g. difficulty-matched phases or an agent-id ablation.

-- The multi-operator teleoperation pipeline extends an existing tool. Human-in-the-loop control inside a shared simulator exists [15], including several users controlling different agents at once [8], and synchronized vr teleoperation of two robots by two operators, one robot being a humanoid, has appeared in recent work [14]. Useful engineering, but not an ml contribution. (The reviewer is aware of closely related concurrent work [16] and does not hold it against the submission.)

-- Table 1 and the related work leave out the line of multi-agent embodied work with visual observations: two agents jointly carrying an object from egocentric pixels [11], human-ai and multi-agent collaboration simulators [6, 8, 15], cooperative agents with costly communication [10], and a visual multi-agent benchmark whose embodiments already include a humanoid, though at the planning level [13]. The claim that visual multi-agent benchmarks have mostly been non-humanoid tabletop ones needs qualifying. Cooperative marl [1, 2, 3, 4] and zero-shot coordination or partner generalization [9, 12] are not engaged with, though they offer standard tests the evaluation could borrow, such as pairing a policy with unseen partners.

-- With no formulation and no clear learning-side finding, in the reading of the reviewer the submission fits a datasets and benchmarks venue better than the main track.

W2. The evaluation does not establish that the benchmark measures collaboration, or how well current methods collaborate.

-- The need for collaboration is not shown. Coupling is enforced through the success criteria (e.g. both robots must hold the object) and not shown to follow from the physics. Object masses and related physical parameters are not reported. By the appendix's own account one task has no collaborative phase at all (though the task as a whole is described as collaborative). There are no controls with a single robot, a scripted or frozen partner, or one robot fewer. Each g1 has two hands, and for several tasks (e.g. pouring from a cup into a held glass, or serving a bottle from a cart) it is not obvious that a second robot is needed.

-- Evaluation tests performance within fixed task setups. Each task uses a fixed scene, fixed robot start poses, object offsets of a few centimeters for most tasks (larger, up to 15 and 30 cm, for trashcollection and cartservice), and one fixed instruction per role, and the paper gives evaluation ranges but not the ranges used when collecting demonstrations. No tests of generalization to new layouts, object instances, partners, role assignments or phrasings are reported, which matters because vlas are known to ignore or memorize instructions in similar settings [17, 18]. There is a single training seed and no uncertainty estimate, and many cells sit between 0 and 5 percent. At 100 rollouts, 5 successes has a 95% interval of roughly 2 to 11 percent, so many of the small per-task gaps in table 2 are hard to interpret.

-- Settings expected to help (more information about the partner, parameter sharing) mostly do not, and the explanations the paper offers (role ambiguity, shift from pretraining) are not tested. Sharing parameters hurts in most of the reported settings, though sharing with an agent index in the observation has long been standard in cooperative marl [5, 7]. For act and dp the shared local setting has no identifier. Giving each robot all robots' views helps dp, is mixed for act, and hurts gr00t n1.7 (in its shared setting this comes together with a switch to the joint instruction). The only learned joint-action method scores below independent per-robot act. No training-fit diagnostic is reported, so underfitting and compounding error cannot be told apart. For gr00t n1.7, the setting used in the main table is also not the best one from the ablation.

-- Both multi-agent baselines were designed and shown on fixed-base arm setups, and the paper does not discuss how adapting them to moving egocentric humanoid cameras affects their results. Their lower scores than act therefore do not by themselves separate limits of the methods from difficulties of the adaptation. The only world action model was developed on tabletop and wheeled dual-arm settings, while a humanoid-oriented world action model and a whole-body controller the paper itself cites [19, 20] are not used or discussed as alternatives. Apart from the fully centralized one, the baselines choose each robot's action separately at execution time, without communication, although the high-coupling tasks are where choosing each agent's action independently is known to be a problem [11].

-- The phase results alone do not separate coordination failures from other ones. On pouring, act and fast-wam pass the lift and tip phases in 84 to 95 percent of rollouts but reach only 1 and 7 percent at the last step. One failure example (fig 8b) shows cup and glass misaligned, but the paper does not report how often failures come from imprecise tipping versus misalignment between the two robots.

Minor concerns

-- The intro text does not name the tasks or the action space, and the decoupled leg controller first appears in sec 3.1 and the appendix. Table 1 defines whole-body at the task level, while the learned policy only commands the upper body and a base velocity and height. Worth making explicit so readers do not assume whole-body control is learned.

-- The phase-level evaluation starts from states taken from the training demonstrations, so fig 4 measures behaviour close to the training data. Worth discussing what this means for the collaborative vs non-collaborative comparison.

-- The llm-agent run uses a different interface, pauses the simulator while reasoning, has a longer time budget and only 10 episodes. Fine as exploratory, but its role in the main text is unclear.

-- Please confirm that every supplementary clip is a direct simulator recording of a policy rollout. A few clips show blur and artifacts that make this hard to tell.

-- Two references have wrong metadata: the pink entry mixes the version, year and author list of different releases, and the humanoidmimicgen title is not the published one. Gr00t n1.7 should also point to its model card, not only the n1 paper.

On exposition: These are smaller points, but they add up. The abstract repeats the same phrase about multi-humanoid collaboration under egocentric observations in consecutive sentences. It then describes the benchmark through general words ("diverse collaboration patterns", "different forms of coordinated perception and control") without saying what the tasks are, what the robot controls, or what makes a task collaborative. Several analysis paragraphs rest on hedges ("one possible explanation", "may reflect", "providing useful insights for future research") without the follow-up experiment that would turn them into findings. Apart from fig 4, the main-text figures are polished but illustrative, and most of the specifics a reader needs (exact action space, phase criteria, instructions) are left to the appendix.

For a benchmark, a new reader should be able to learn from the first two pages what the robots control, which tasks need more than one robot, and what a 10 percent success rate tells them. That takes deliberate exposition and a clear flow from problem to design to evidence, and in the reading of the reviewer this onus is largely unmet here. The positioning in particular would benefit from a rewrite grounded in the multi-agent and embodied ai literature.

Questions for authors
  1. (W2, need for collaboration) Which tasks can not be completed by one robot in the simulator, and what prevents it (object mass relative to the g1 payload, size, reach)? On a few high-coupling tasks, results with a single robot, or with the partner replaced by a scripted or frozen one, would show whether success depends on the partner's behaviour and not only on the success criteria. Reporting object masses would also help.

  2. (W2, evaluation scope and rigor) Do the main conclusions hold beyond the fixed task setups and across training runs? Perturbations at evaluation time need no retraining: larger pose offsets, a different start pose, rephrased instructions, or swapped roles. Several seeds with intervals on a representative subset would show which gaps in table 2 are real. A training-fit diagnostic (success from training intial conditions, or train vs validation action error) would help assess whether fitting problems contribute to the poor rollout performance.

  3. (W2, upper bounds and baselines) Do the paper's own explanations for table 3 hold up? Adding an agent identifier to the shared act/dp setting directly tests the role-ambiguity explanation. How were latenttom and gaudp adapted to moving egocentric cameras? Is there any setting where giving a robot more information about its partner (shared views, joint prediction, an identifier) clearly helps on the high-coupling tasks? If none does, the benchmark may not yet reward collaboration.

  4. (W1, positioning and contribution) What learning question does the benchmark isolate that prior multi-agent embodied benchmarks do not? A short formal statement of the intended setting (what each robot observes, what can be shared, and what is centralized at training vs execution) would help, together with one conclusion about learning collaborative policies that the experiments support. How does the teleoperation pipeline differ in capability from existing multi-user and multi-operator setups?

Q1 and Q2 matter most for the reviewer's assessment.


References

[1] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (maddpg). NeurIPS 2017.

[2] QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. ICML 2018.

[3] MAVEN: Multi-Agent Variational Exploration. NeurIPS 2019.

[4] The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (mappo). NeurIPS 2022 Datasets and Benchmarks.

[5] Cooperative Multi-agent Control Using Deep Reinforcement Learning. AAMAS 2017 workshops.

[6] Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration. ICLR 2021.

[7] Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning. arXiv 2020.

[8] PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks. ICLR 2025.

[9] "Other-Play" for Zero-Shot Coordination. ICML 2020.

[10] Building Cooperative Embodied Agents Modularly with Large Language Models (CoELA). ICLR 2024.

[11] A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks. ECCV 2020.

[12] Collaborating with Humans without Human Data (fcp). NeurIPS 2021.

[13] VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning (VIKI-Bench). NeurIPS 2025 Datasets and Benchmarks.

[14] DUET: Dual-Robot Understanding via Efficient Teaching. arXiv 2026.

[15] Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots. ICLR 2024.

[16] MATE: Multi-Agent Virtual Teleoperation Platform for Humanoid Collaboration Data Collection. arXiv 2026.

[17] LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization. arXiv 2025.

[18] LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. arXiv 2025.

[19] MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulation. arXiv 2026.

[20] SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control. Science Robotics 2026.

10 comments
AuthorsOpenReview ↗

Response to Reviewer vFKa (1/9): Q1

Thank you for the detailed and constructive review. Following your suggestions, we have run additional experiments with a frozen partner, wider placement ranges, three training seeds, an agent identifier and a late-starting partner, and clarified the learning question of the benchmark. We answer Q1-Q4 first and then the remaining weaknesses. Section, table and figure numbers below follow the revised paper, and "the original Table 2" refers to Table 2 of the initial submission (seed 0).

  • "Which tasks can not be completed by one robot in the simulator, and what prevents it (object mass relative to the g1 payload, size, reach)?"

High physical coupling tasks (FrameHang, CoPouring, CoCarry, TableAlign, and the three-humanoid tasks MoveHouse and BigTable). These tasks cannot be done by one robot. The robots must lift and move a large, heavy object while keeping it level, for example a 1.15 m wide framed painting (4 kg) or a 1.4 m long table (4 kg). A single robot can hold only one side of the large object, so it cannot lift the object or keep it level, and the task fails.

Low physical coupling tasks (Handover, Pouring, TrashCollection, CartService). A single robot can complete these tasks if given enough time, but not efficiently. It would have to do both robots' work in sequence, which takes more time and more walking. Dividing the work between two robots avoids this. Handover and Pouring are designed as tasks with low base movement. The objects or the target position lie in front of different robots, on opposite sides of a counter or table. With a single robot, these tasks would require high base movement, because the robot would have to walk around the counter or table. TrashCollection and CartService already have high base movement with two robots. A single robot would have to carry the bin and then sweep, or push the trolley and then serve, walking back and forth between the work areas. Collaboration is therefore needed for efficiency rather than for feasibility. These tasks test whether policies can carry out such a division of labor, which is needed to scale collaboration to larger numbers of robots and more diverse tasks.

We have added representative task examples to the Introduction section to clarify this distinction.

  • "On a few high-coupling tasks, results with a single robot, or with the partner replaced by a scripted or frozen one, would show whether success depends on the partner's behaviour and not only on the success criteria."

Following your suggestion, we evaluate our trained ACT policy on FrameHang and TableAlign with one robot frozen at its initial pose. While the phase criteria in the paper require both robots to hold the object, we change them to require only the moving robot to hold it and keep every requirement on the object unchanged. The table below reports the phase-wise success rates (%) with both robots active (Original) and with one robot frozen.

Original p1p_1Original p2p_2Original p3p_3One robot frozen p1p_1One robot frozen p2p_2One robot frozen p3p_3
FrameHang1001004410020
TableAlign100793910000

Task success (p3p_3) drops to 0% on both tasks. The moving robot still completes p1p_1 in every rollout, grasping its side of the frame or lifting its end of the table. It then fails at p2p_2, lifting the frame or carrying the table to the goal. Success therefore depends on the partner's behavior, not only on the success criteria. We have added this experiment to Sec. 4.4 and Appendix F.2.

  • "Reporting object masses would also help."

We have added the mass of every task object to Table 6. For example, the laundry basket in CoCarry weighs 3 kg, and the round table in BigTable weighs 7.5 kg, while the wine bottle in Handover weighs 1.5 kg.

We continue our rebuttal in the next comment.

AuthorsOpenReview ↗

Response to Reviewer vFKa (2/9): Q2 (part 1/2)

  • "Do the main conclusions hold beyond the fixed task setups and across training runs? Perturbations at evaluation time need no retraining: larger pose offsets, a different start pose, rephrased instructions, or swapped roles."

Among the perturbations you suggested, we test larger pose offsets as a representative case. Without retraining, we evaluate all policies in Table 2 on object and target placements outside the evaluation ranges in Table 6. Table 15 in the appendix lists the wider range of each task. For example, the basket in CoCarry is placed up to 0.10 m from its nominal position (0.05 m in Table 6), and its goal shelf, fixed in the original evaluation, moves by 0.30-0.40 m.

The table below reports the phase-wise success rates (%) of each policy under the original and wider ranges, averaged over the two-humanoid tasks, where p3p_3 is task success.

Original ranges p1p_1Original ranges p2p_2Original ranges p3p_3Wider ranges p1p_1Wider ranges p2p_2Wider ranges p3p_3
ACT81.065.427.955.328.62.6
DP63.633.910.440.316.01.1
LatentToM59.022.05.337.37.30.3
GauDP63.135.113.441.015.10.5
π0.5\pi_{0.5}63.325.94.536.412.60.6
GR00T N1.761.436.39.437.521.00.8
Ψ0\Psi_055.412.41.037.35.90.4
Fast-WAM79.051.812.149.925.62.1

Success drops considerably for every policy while the overall trend is largely preserved. ACT still performs best among the standard and multi-agent IL policies and Fast-WAM among the VLAs and the WAM at every phase. We have added these results to Sec. 4.2 and Appendix F.1.

  • "Several seeds with intervals on a representative subset would show which gaps in table 2 are real."

Following your suggestion, we retrain ACT, DP, the three VLAs, and Fast-WAM with two additional seeds (three seeds in total) and evaluate them on the two-humanoid tasks. Due to computational costs and the limited rebuttal period, the three-seed results of LatentToM and GauDP and of the three-humanoid tasks will be included in the final version. The table below reports the task success rates (%) as mean ±\pm standard deviation over the three seeds, with 100 rollouts per task and seed. Its last column gives the average in the original Table 2 (seed 0), and the revised Table 2 includes the three-seed results.

HandoverPouringFrameHangCoPouringTrashCollectionCartServiceCoCarryTableAlignAvg.Original Avg.
ACT55.0$\pm$9.21.0$\pm$1.046.7$\pm$2.59.7$\pm$1.20.3$\pm$0.61.7$\pm$1.556.7$\pm$4.043.7$\pm$5.026.8$\pm$1.127.9
DP26.7$\pm$3.10.0$\pm$0.024.3$\pm$1.55.3$\pm$2.10.3$\pm$0.62.3$\pm$0.613.7$\pm$1.53.7$\pm$0.69.5$\pm$0.810.4
π0.5\pi_{0.5}2.3$\pm$0.60.0$\pm$0.023.7$\pm$4.90.3$\pm$0.60.0$\pm$0.03.3$\pm$1.21.3$\pm$0.61.3$\pm$0.64.0$\pm$0.84.5
GR00T N1.722.7$\pm$6.70.3$\pm$0.633.3$\pm$10.11.0$\pm$1.70.3$\pm$0.60.0$\pm$0.01.7$\pm$0.60.3$\pm$0.67.5$\pm$2.29.4
Ψ0\Psi_00.7$\pm$1.20.0$\pm$0.09.0$\pm$5.31.0$\pm$1.72.3$\pm$1.20.0$\pm$0.00.3$\pm$0.60.3$\pm$0.61.7$\pm$1.11.0
Fast-WAM4.7$\pm$3.111.7$\pm$6.428.0$\pm$1.016.3$\pm$13.81.3$\pm$1.51.0$\pm$1.024.3$\pm$9.113.3$\pm$4.712.6$\pm$1.512.1

The gaps between the average success rates in the original Table 2 hold across seeds. ACT remains clearly above DP, and Fast-WAM remains the best among the VLAs and the WAM.

We continue our rebuttal in the next comment.

AuthorsOpenReview ↗

Response to Reviewer vFKa (3/9): Q2 (part 2/2)

  • "A training-fit diagnostic (success from training intial conditions, or train vs validation action error) would help assess whether fitting problems contribute to the poor rollout performance."

As suggested, we report the training and validation losses of the VLAs and the WAM (seed 0), as logged during training. The validation loss is computed on the 10 demonstrations per task that are excluded from training. The table below lists both losses from 10k to 40k training steps, with the training loss averaged over 1,000 steps. Each model uses its own training objective, so the values are not comparable across models.

Loss10k20k30k40k
π0.5\pi_{0.5}Training0.06510.04600.03580.0324
π0.5\pi_{0.5}Validation0.07120.05950.05560.0576
GR00T N1.7Training0.03790.02560.02000.0182
GR00T N1.7Validation0.03120.02540.02370.0217
Ψ0\Psi_0Training2.501.431.050.95
Ψ0\Psi_0Validation2.311.280.981.05
Fast-WAMTraining0.2400.2120.1970.192
Fast-WAMValidation0.2710.2550.2510.254

Training and validation losses decrease overall and generally approach a plateau by 40k steps, with no marked increase in validation loss. These trends support our choice of 40k steps as a reasonable training budget. We additionally trained GR00T N1.7 with a new seed (seed 3) and evaluated its checkpoints at 10k, 20k, 30k and 40k training steps. The table below reports its task success rates (%) with 100 rollouts per task and checkpoint. The last row gives the mean ±\pm standard deviation over seeds 0-2 at 40k steps for reference.

HandoverPouringFrameHangCoPouringTrashCollectionCartServiceCoCarryTableAlignAvg.
10k (seed 3)000000000.0
20k (seed 3)102200000.6
30k (seed 3)11333100116.3
40k (seed 3)17032100206.5
40k (seeds 0-2)22.7$\pm$6.70.3$\pm$0.633.3$\pm$10.11.0$\pm$1.70.3$\pm$0.60.0$\pm$0.01.7$\pm$0.60.3$\pm$0.67.5$\pm$2.2

As shown in the table above, average task success reaches 6.3% at 30k and changes little at 40k (6.5%). These results suggest that the low final success in this run is not simply due to evaluating the policy too early in training.

We continue our rebuttal in the next comment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful and constructive feedback. We have revised the paper accordingly and marked the changes in blue. The major changes are summarized below.

Additional experiments

  • Three training seeds (Table 2). We retrain ACT, DP, the three VLAs and Fast-WAM with two additional seeds and report the mean and standard deviation on the two-humanoid tasks.
  • Frozen partner (Sec. 4.4, Table 3 and Appendix F.2). With one robot frozen at its initial pose, ACT no longer succeeds on FrameHang and TableAlign.
  • Partner timing (Sec. 4.4, Fig. 5 and Appendix F.3). When one robot starts late or pauses early in the episode, the success of GR00T N1.7 on the two-humanoid tasks drops sharply.
  • Wider placement ranges (Sec. 4.2 and Appendix F.1). With object and target placements outside the original evaluation ranges, success drops for every policy, while ACT and Fast-WAM remain the best in their groups.
  • Agent identifier (Sec. 4.5 and Appendix F.4). A one-hot agent identifier slightly improves the average success of the shared ACT and DP policies.

Clarifications

  • Learning question and positioning (Introduction and Related Work sections). We state the learning question and the main finding, and discuss prior multi-agent embodied benchmarks and cooperative multi-agent reinforcement learning.
  • Problem setting (Introduction section and Sec. 3.1). We state the action interface of the learned policies and the observation assumptions.
  • Physical parameters (Appendix A, Table 5 and Table 6). We report the mass of every task object and the finger torque limits and friction of the hands, and discuss how the hand settings differ from the real hardware.
  • Abstract and exposition (Abstract, Sec. 4.4 and Sec. 4.5). We remove repeated phrases from the abstract and name concrete tasks. We also replace tentative explanations with experimental evidence where we could test them.

The supplementary material now contains our code. Due to the 100 MB size limit, the videos of the initial submission are no longer included, but they can still be downloaded from the earlier Supplementary Material under Revisions.

Section, table and figure numbers in our responses follow the revised paper. We respond to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.