W1. The submission reads as a dataset and systems effort without a clear learning question, and its related work leaves out the visual multi-agent and cooperative marl work closest to it.
-- There is no formal problem statement. The paper does not define a dec-pomdp or similar, does not state a communication model (explicit inter-agent communication is only mentioned as future work), and does not state a sensing-noise model or localization assumptions. The baselines run under quite different regimes (fully decentralized, centralized training with decentralized execution plus a shared third-person view, fully centralized), so it is unclear which learning problem the benchmark is meant to pose. With no sim-to-real evidence, the case for the benchmark has to come from what it reveals about learning methods, and the analysis beyond success rates (phase-level results, qualitative failures, one ablation) does not yet isolate a cause on the learning or coordination side.
-- Sec 4.3 finds collaborative phases harder, but the authors note this may reflect phase difficulty. Sec 4.4 concludes that the effects of observation scope and sharing vary across models. Neither result is followed by an experiment that isolates the cause, e.g. difficulty-matched phases or an agent-id ablation.
-- The multi-operator teleoperation pipeline extends an existing tool. Human-in-the-loop control inside a shared simulator exists [15], including several users controlling different agents at once [8], and synchronized vr teleoperation of two robots by two operators, one robot being a humanoid, has appeared in recent work [14]. Useful engineering, but not an ml contribution. (The reviewer is aware of closely related concurrent work [16] and does not hold it against the submission.)
-- Table 1 and the related work leave out the line of multi-agent embodied work with visual observations: two agents jointly carrying an object from egocentric pixels [11], human-ai and multi-agent collaboration simulators [6, 8, 15], cooperative agents with costly communication [10], and a visual multi-agent benchmark whose embodiments already include a humanoid, though at the planning level [13]. The claim that visual multi-agent benchmarks have mostly been non-humanoid tabletop ones needs qualifying. Cooperative marl [1, 2, 3, 4] and zero-shot coordination or partner generalization [9, 12] are not engaged with, though they offer standard tests the evaluation could borrow, such as pairing a policy with unseen partners.
-- With no formulation and no clear learning-side finding, in the reading of the reviewer the submission fits a datasets and benchmarks venue better than the main track.
W2. The evaluation does not establish that the benchmark measures collaboration, or how well current methods collaborate.
-- The need for collaboration is not shown. Coupling is enforced through the success criteria (e.g. both robots must hold the object) and not shown to follow from the physics. Object masses and related physical parameters are not reported. By the appendix's own account one task has no collaborative phase at all (though the task as a whole is described as collaborative). There are no controls with a single robot, a scripted or frozen partner, or one robot fewer. Each g1 has two hands, and for several tasks (e.g. pouring from a cup into a held glass, or serving a bottle from a cart) it is not obvious that a second robot is needed.
-- Evaluation tests performance within fixed task setups. Each task uses a fixed scene, fixed robot start poses, object offsets of a few centimeters for most tasks (larger, up to 15 and 30 cm, for trashcollection and cartservice), and one fixed instruction per role, and the paper gives evaluation ranges but not the ranges used when collecting demonstrations. No tests of generalization to new layouts, object instances, partners, role assignments or phrasings are reported, which matters because vlas are known to ignore or memorize instructions in similar settings [17, 18]. There is a single training seed and no uncertainty estimate, and many cells sit between 0 and 5 percent. At 100 rollouts, 5 successes has a 95% interval of roughly 2 to 11 percent, so many of the small per-task gaps in table 2 are hard to interpret.
-- Settings expected to help (more information about the partner, parameter sharing) mostly do not, and the explanations the paper offers (role ambiguity, shift from pretraining) are not tested. Sharing parameters hurts in most of the reported settings, though sharing with an agent index in the observation has long been standard in cooperative marl [5, 7]. For act and dp the shared local setting has no identifier. Giving each robot all robots' views helps dp, is mixed for act, and hurts gr00t n1.7 (in its shared setting this comes together with a switch to the joint instruction). The only learned joint-action method scores below independent per-robot act. No training-fit diagnostic is reported, so underfitting and compounding error cannot be told apart. For gr00t n1.7, the setting used in the main table is also not the best one from the ablation.
-- Both multi-agent baselines were designed and shown on fixed-base arm setups, and the paper does not discuss how adapting them to moving egocentric humanoid cameras affects their results. Their lower scores than act therefore do not by themselves separate limits of the methods from difficulties of the adaptation. The only world action model was developed on tabletop and wheeled dual-arm settings, while a humanoid-oriented world action model and a whole-body controller the paper itself cites [19, 20] are not used or discussed as alternatives. Apart from the fully centralized one, the baselines choose each robot's action separately at execution time, without communication, although the high-coupling tasks are where choosing each agent's action independently is known to be a problem [11].
-- The phase results alone do not separate coordination failures from other ones. On pouring, act and fast-wam pass the lift and tip phases in 84 to 95 percent of rollouts but reach only 1 and 7 percent at the last step. One failure example (fig 8b) shows cup and glass misaligned, but the paper does not report how often failures come from imprecise tipping versus misalignment between the two robots.
Minor concerns
-- The intro text does not name the tasks or the action space, and the decoupled leg controller first appears in sec 3.1 and the appendix. Table 1 defines whole-body at the task level, while the learned policy only commands the upper body and a base velocity and height. Worth making explicit so readers do not assume whole-body control is learned.
-- The phase-level evaluation starts from states taken from the training demonstrations, so fig 4 measures behaviour close to the training data. Worth discussing what this means for the collaborative vs non-collaborative comparison.
-- The llm-agent run uses a different interface, pauses the simulator while reasoning, has a longer time budget and only 10 episodes. Fine as exploratory, but its role in the main text is unclear.
-- Please confirm that every supplementary clip is a direct simulator recording of a policy rollout. A few clips show blur and artifacts that make this hard to tell.
-- Two references have wrong metadata: the pink entry mixes the version, year and author list of different releases, and the humanoidmimicgen title is not the published one. Gr00t n1.7 should also point to its model card, not only the n1 paper.
On exposition: These are smaller points, but they add up. The abstract repeats the same phrase about multi-humanoid collaboration under egocentric observations in consecutive sentences. It then describes the benchmark through general words ("diverse collaboration patterns", "different forms of coordinated perception and control") without saying what the tasks are, what the robot controls, or what makes a task collaborative. Several analysis paragraphs rest on hedges ("one possible explanation", "may reflect", "providing useful insights for future research") without the follow-up experiment that would turn them into findings. Apart from fig 4, the main-text figures are polished but illustrative, and most of the specifics a reader needs (exact action space, phase criteria, instructions) are left to the appendix.
For a benchmark, a new reader should be able to learn from the first two pages what the robots control, which tasks need more than one robot, and what a 10 percent success rate tells them. That takes deliberate exposition and a clear flow from problem to design to evidence, and in the reading of the reviewer this onus is largely unmet here. The positioning in particular would benefit from a rewrite grounded in the multi-agent and embodied ai literature.
Response to Reviewer 2yY5 (1/2): W1, W2, W3
Thank you for the positive assessment and constructive feedback. Following your suggestions, we have added our code to the supplementary material and report results over three training seeds. Section, table and figure numbers below follow the revised paper, and "the original Table 2" refers to Table 2 of the initial submission (seed 0). Please find our response below.
We have added our code to the supplementary material. We ran all training and evaluation on a mix of NVIDIA RTX 3090, RTX 4090, RTX A6000 and RTX PRO 6000 GPUs. For larger models such as Fast-WAM, we used high-memory GPUs such as the RTX PRO 6000 so that each training run fits on a single GPU.
Following your suggestion, we retrain ACT, DP, the three VLAs, and Fast-WAM with two additional seeds (three seeds in total) and evaluate them on the two-humanoid tasks. Due to computational costs and the limited rebuttal period, the three-seed results of LatentToM and GauDP and of the three-humanoid tasks will be included in the final version. The table below reports the task success rates (%) as mean ± standard deviation over the three seeds, with 100 rollouts per task and seed. Its last column gives the average in the original Table 2 (seed 0), and the revised Table 2 includes the three-seed results.
The gaps between the average success rates in the original Table 2 hold across seeds. ACT remains clearly above DP, and Fast-WAM remains the best among the VLAs and the WAM.
CoHuB is a benchmark contribution. ICLR has published many dataset and benchmark papers, including RoboCasa365 [1] and OGBench [2], as well as multi-agent embodied benchmarks such as Habitat 3.0 [3] and PARTNR [4]. Beyond the benchmark and dataset, our analyses focus on collaboration rather than on robustness to scenes, objects or instructions. We study where collaborative phases fail and how role information and parameter sharing affect multi-humanoid policies. These analyses explore how vision-based policies collaborate with each other, which remains largely underexplored.
References
[1] RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots. ICLR 2026.
[2] OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025.
[3] Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots. ICLR 2024.
[4] PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks. ICLR 2025.
We continue our rebuttal in the next comment.