- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes HRC-SimBench, a simulation benchmark for robot policies that work with a person. An LLM writes a script for each task (human role, timed steps, ordering rules, success conditions), HY-Motion 1.0 generates the human motion, and filtering plus a manual audit keep 786 of 2,000 generated trajectories. The 13 tasks cover three roles (Instructor, Collaborator, Interferer), with 50 training and 10 test episodes each. Models such as GR00T N1.5, π0.5 and ACT are evaluated on the proposed benchmark. On a real SO-100 arm, first adapting on the simulated data raises success from 3/30 to 13/30.
Critical strengths
- Important problem. Most VLA benchmarks have no person in the loop, and making the human’s role part of the task should make failures easier to interpret.
- The evaluation protocol is clearly specified (thresholds in Table 5, failure handling in Tables 6-7), and Interferer perturbations are replayed identically for every model.
- The data pipeline is documented, with filtering counts (Fig. 7, Table 13) and a double-annotated audit.
Critical weaknesses
- The simulated person does not react to the robot. The human motion is generated before the rollout and replayed, with Interferer events started by triggers, so the benchmark tests working around a scripted partner rather than “coordination under shared agency” as stated in abstract.
- The central claims aren’t tested. Without human-free versions of the tasks, a low CSR may reflect weak manipulation or too little adaptation data rather than poor collaboration, so the main hypothesis (l. 294-295) stays open. No zero-shot result backs the claim that fine-tuning “consistently improves collaborative performance”. Most model gaps are within noise (GR00T N1.5 vs ACT on Collaborator tasks: 35/70 vs 26/70, Fisher p ≈ 0.17), so the claim that VLA pretraining explains the gap (l. 374-375) isn’t supported.
- The simulated person seems to be a replayed animation, and it’s unclear how it fits the scene. The motion is generated before the rollout and Interferer events start on triggers, so the person never reacts to the robot. HY-Motion 1.0 takes only text and a duration and has no finger joints, yet the tasks include handovers, taking a block from a hand (C4) and carrying a pot together (C2). The refinement meant to align the motion with these targets is described only in general terms and the only motion check is wrist speed.
- The evaluation set is smaller than stated. The “over 650 evaluation episodes” are the adaptation split. The held-out evaluation set has 130 episodes (Table 10), only 30 of them on Interferer tasks.
- Positioning. The components are existing techniques, several metrics are close to Hoffman’s fluency measures (IEEE THMS 2019, not cited), and Table 1 leaves out related work such as HumanTHOR, Assistive Gym, RCareWorld and GenH2R.
Questions for authors
- Does the person react to the robot beyond triggers? How are the HY-Motion clips placed and refined, and how are hand-object contacts (C2, C4) simulated?
- The idle ratio (Eq. 4) is never reported, the Safety column (0.98-1.00 on Instructor and Collaborator tasks) doesn’t separate the models, and several appendix references point to the wrong main-paper tables (e.g., l. 816, 1449-1453).
2 comments
Kind Reminder to Reviewer FVNU
Dear Reviewer FVNU,
Thank you again for your thorough review. The discussion period is about to close, so we would like to check in briefly. We have replied to each of your points and revised the PDF accordingly (changes in blue). If our response has resolved your concerns, we would greatly appreciate a short reply or an updated assessment. If anything is still unclear, please let us know and we will answer right away.
Best regards, The Authors
Response to Reviewer FVNU
Thank you for the thorough review. Your comments have helped us improve the paper, and we have incorporated them in the revised PDF. For several points, the supporting evidence is already in the paper, and we point to it below.
The person does not react to the robot. Thank you for the suggestion. We have made this scope explicit in the paper. This is a deliberate trade-off, much like a crash-test dummy: every policy faces exactly the same human, so differences in outcome reflect the policy and not the partner. The loop is still closed at the event level. The robot must wait for human readiness (a stable gesture, a hand inside the handover zone), and interventions are triggered by the robot's own state (e.g., the takeover begins as the robot approaches the pad; App. C, Listing 3). The Introduction now states this scope directly: the benchmark "measures how a policy responds to given human cues, motion, and interventions, not two-way adaptation between partners." We also added a Limitations paragraph.
Manipulation or collaboration?Three results separate the two. First, Table 27 uses the same robot and scenes: training on object-only data, which teaches the manipulation, yields CSR 0.133 on Collaborator and 0.000 on Interferer, versus 0.500 and 0.100 with interaction data. Manipulation skill alone does not remove the failures. Second, the policies can execute the steps: OC is 1.000 on Collaborator and Interferer tasks, and Collaborator Safety is at least 0.98. Performance collapses only when a person intervenes. Third, the failure causes are interactive (Table 25): misreading or pre-empting the cue, lagging the partner, and failing to yield or recover. Sec. 5.3 now summarizes this ablation. The sentence about fine-tuning referred to Table 27 and to Sim+Real vs. Real-only (Table 4), and we have reworded it in the abstract to say so.
Model differences. Thank you for the suggestion. Our central finding is the gap between roles, which is large and statistically robust: across models, 93/210 Collaborator successes vs. 9/90 Interferer successes (Fisher p ≈ 2×10⁻⁹). The real-robot improvement from 3/30 to 13/30 is also significant (p = 0.007). To keep the focus on these findings, we have revised the discussion of VLA pretraining, since the GR00T vs. ACT difference on Collaborator tasks (35/70 vs. 26/70) does not reach significance at this sample size.
Evaluation size. Thank you for the suggestion. The original wording was ambiguous, and we have corrected it throughout: the split is 650 training and 130 held-out evaluation episodes (Table 10).
Hands and contact. Kinematic attachment is a common choice in embodied and HRI simulators such as Habitat 3.0, PARTNR, HandoverSim, and BEHAVIOR-1K. Following this practice, objects are attached to the human hand rather than grasped by simulated fingers; the benchmark evaluates the robot, not human dexterity. Attachment offsets are tuned per object so the object sits naturally in the hand. Script events decide when it attaches and releases. In C4, the block stays in the person's hand until the gripper closes on it inside the 12 cm handover zone. In C3, the person takes the object once their hand has been still in the zone for 0.5 s and the gripper opens. In C2, one side of the pot follows the human hand, so if the robot lags, the pot slips or tilts, which TSync and CSR capture. Thank you for pointing this out. The mocap comparison in Sec. 5.2 is computed on wrist-speed profiles (peak velocity, duration, and jerk). Two further checks cover the whole interaction rather than the wrist alone: a human–object consistency check rejects clips with penetration or inconsistent contact (196 of 2,000), and a manual semantic audit removes 148 more. These details are now in Sec. 4 and App. D.
Related work. IR and CT are adapted from Hoffman's fluency metrics. We now cite IEEE THMS 2019 in Sec. 3 and discuss HumanTHOR, Assistive Gym, RCareWorld, and GenH2R in Sec. 2 and Table 1.
IR and Safety.Thank you for catching this. We have added IR to Table 3. For GR00T and ACT, IR is 0.043 and 0.105 on Instructor, 0.111 and 0.102 on Collaborator, and 0.059 and 0.043 on Interferer. Interferer IR is low while CSR is also low, so these robots do not fail by freezing; they fail by pressing on with the original plan. Safety averages the safety checks that apply to each role. It stays near 1 with cooperative partners and separates the models once the person actively intrudes (Interferer: 0.735 / 0.729 / 0.721). The Table 3 caption now lists which components Safety averages for each role, and the cross-references are fixed.