acceptodds
Under review as a conference paper at ICLR 2027

HRC-SimBench: Benchmarking Interaction-Centric Human-Robot Collaboration

Abstract

Current vision-language-action (VLA) benchmarks primarily evaluate isolated manipulation skills while leaving human-robot interaction structure largely unmodeled. However, real-world collaboration fundamentally requires coordination under shared agency, including intent understanding, temporal synchronization, protocol adherence, and safe interaction in dynamic environments. To address this gap, we introduce HRC-SimBench, a diagnostic benchmark for intent-aware human-robot collaboration based on executable interaction scenarios. HRC-SimBench represents collaborative tasks as structured scenario scripts that explicitly model agent roles, temporal dependencies, coordination constraints, and human behavior distributions. Building on this abstraction, HRC-SimBench defines three representative interaction roles: Instructor, Collaborator, and Intruder, covering intent communication, joint coordination, and robustness under human intervention. The benchmark contains 13 role-conditioned tasks with over 650 evaluation episodes generated from diverse interaction trajectories and scene variations. Beyond binary task success, HRC-SimBench introduces interpretable interaction-centric metrics spanning synchronization, responsiveness, protocol compliance, and safety. We evaluate adapted policies based on GR00T, π0.5, and ACT under a unified protocol. Results show that current foundation robot policies struggle substantially in collaborative settings despite strong manipulation ability, revealing major limitations in temporal coordination and intent-aware behavior. Fine-tuning on HRC-SimBench consistently improves collaborative performance. In a real-world adaptation study, simulation data generated by HRC-SimBench improves GR00T N1.5’s physical-task success rate from 0.10 to 0.43, demonstrating the benchmark’s value for advancing interaction-centric robot learning.

Primary area:
datasets and benchmarks
Keywords:
Human-robot interaction and natural language instruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FVNUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HRC-SimBench, a simulation benchmark for robot policies that work with a person. An LLM writes a script for each task (human role, timed steps, ordering rules, success conditions), HY-Motion 1.0 generates the human motion, and filtering plus a manual audit keep 786 of 2,000 generated trajectories. The 13 tasks cover three roles (Instructor, Collaborator, Interferer), with 50 training and 10 test episodes each. Models such as GR00T N1.5, π0.5 and ACT are evaluated on the proposed benchmark. On a real SO-100 arm, first adapting on the simulated data raises success from 3/30 to 13/30.

Critical strengths
  • Important problem. Most VLA benchmarks have no person in the loop, and making the human’s role part of the task should make failures easier to interpret.
  • The evaluation protocol is clearly specified (thresholds in Table 5, failure handling in Tables 6-7), and Interferer perturbations are replayed identically for every model.
  • The data pipeline is documented, with filtering counts (Fig. 7, Table 13) and a double-annotated audit.
Critical weaknesses
  • The simulated person does not react to the robot. The human motion is generated before the rollout and replayed, with Interferer events started by triggers, so the benchmark tests working around a scripted partner rather than “coordination under shared agency” as stated in abstract.
  • The central claims aren’t tested. Without human-free versions of the tasks, a low CSR may reflect weak manipulation or too little adaptation data rather than poor collaboration, so the main hypothesis (l. 294-295) stays open. No zero-shot result backs the claim that fine-tuning “consistently improves collaborative performance”. Most model gaps are within noise (GR00T N1.5 vs ACT on Collaborator tasks: 35/70 vs 26/70, Fisher p ≈ 0.17), so the claim that VLA pretraining explains the gap (l. 374-375) isn’t supported.
  • The simulated person seems to be a replayed animation, and it’s unclear how it fits the scene. The motion is generated before the rollout and Interferer events start on triggers, so the person never reacts to the robot. HY-Motion 1.0 takes only text and a duration and has no finger joints, yet the tasks include handovers, taking a block from a hand (C4) and carrying a pot together (C2). The refinement meant to align the motion with these targets is described only in general terms and the only motion check is wrist speed.
  • The evaluation set is smaller than stated. The “over 650 evaluation episodes” are the adaptation split. The held-out evaluation set has 130 episodes (Table 10), only 30 of them on Interferer tasks.
  • Positioning. The components are existing techniques, several metrics are close to Hoffman’s fluency measures (IEEE THMS 2019, not cited), and Table 1 leaves out related work such as HumanTHOR, Assistive Gym, RCareWorld and GenH2R.
Questions for authors
  • Does the person react to the robot beyond triggers? How are the HY-Motion clips placed and refined, and how are hand-object contacts (C2, C4) simulated?
  • The idle ratio (Eq. 4) is never reported, the Safety column (0.98-1.00 on Instructor and Collaborator tasks) doesn’t separate the models, and several appendix references point to the wrong main-paper tables (e.g., l. 816, 1449-1453).
2 comments
AuthorsOpenReview ↗

Response to Reviewer FVNU

Thank you for the thorough review. Your comments have helped us improve the paper, and we have incorporated them in the revised PDF. For several points, the supporting evidence is already in the paper, and we point to it below.

  1. The person does not react to the robot. Thank you for the suggestion. We have made this scope explicit in the paper. This is a deliberate trade-off, much like a crash-test dummy: every policy faces exactly the same human, so differences in outcome reflect the policy and not the partner. The loop is still closed at the event level. The robot must wait for human readiness (a stable gesture, a hand inside the handover zone), and interventions are triggered by the robot's own state (e.g., the takeover begins as the robot approaches the pad; App. C, Listing 3). The Introduction now states this scope directly: the benchmark "measures how a policy responds to given human cues, motion, and interventions, not two-way adaptation between partners." We also added a Limitations paragraph.

  2. Manipulation or collaboration?Three results separate the two. First, Table 27 uses the same robot and scenes: training on object-only data, which teaches the manipulation, yields CSR 0.133 on Collaborator and 0.000 on Interferer, versus 0.500 and 0.100 with interaction data. Manipulation skill alone does not remove the failures. Second, the policies can execute the steps: OC is 1.000 on Collaborator and Interferer tasks, and Collaborator Safety is at least 0.98. Performance collapses only when a person intervenes. Third, the failure causes are interactive (Table 25): misreading or pre-empting the cue, lagging the partner, and failing to yield or recover. Sec. 5.3 now summarizes this ablation. The sentence about fine-tuning referred to Table 27 and to Sim+Real vs. Real-only (Table 4), and we have reworded it in the abstract to say so.

  3. Model differences. Thank you for the suggestion. Our central finding is the gap between roles, which is large and statistically robust: across models, 93/210 Collaborator successes vs. 9/90 Interferer successes (Fisher p ≈ 2×10⁻⁹). The real-robot improvement from 3/30 to 13/30 is also significant (p = 0.007). To keep the focus on these findings, we have revised the discussion of VLA pretraining, since the GR00T vs. ACT difference on Collaborator tasks (35/70 vs. 26/70) does not reach significance at this sample size.

  4. Evaluation size. Thank you for the suggestion. The original wording was ambiguous, and we have corrected it throughout: the split is 650 training and 130 held-out evaluation episodes (Table 10).

  5. Hands and contact. Kinematic attachment is a common choice in embodied and HRI simulators such as Habitat 3.0, PARTNR, HandoverSim, and BEHAVIOR-1K. Following this practice, objects are attached to the human hand rather than grasped by simulated fingers; the benchmark evaluates the robot, not human dexterity. Attachment offsets are tuned per object so the object sits naturally in the hand. Script events decide when it attaches and releases. In C4, the block stays in the person's hand until the gripper closes on it inside the 12 cm handover zone. In C3, the person takes the object once their hand has been still in the zone for 0.5 s and the gripper opens. In C2, one side of the pot follows the human hand, so if the robot lags, the pot slips or tilts, which TSync and CSR capture. Thank you for pointing this out. The mocap comparison in Sec. 5.2 is computed on wrist-speed profiles (peak velocity, duration, and jerk). Two further checks cover the whole interaction rather than the wrist alone: a human–object consistency check rejects clips with penetration or inconsistent contact (196 of 2,000), and a manual semantic audit removes 148 more. These details are now in Sec. 4 and App. D.

  6. Related work. IR and CT are adapted from Hoffman's fluency metrics. We now cite IEEE THMS 2019 in Sec. 3 and discuss HumanTHOR, Assistive Gym, RCareWorld, and GenH2R in Sec. 2 and Table 1.

  7. IR and Safety.Thank you for catching this. We have added IR to Table 3. For GR00T and ACT, IR is 0.043 and 0.105 on Instructor, 0.111 and 0.102 on Collaborator, and 0.059 and 0.043 on Interferer. Interferer IR is low while CSR is also low, so these robots do not fail by freezing; they fail by pressing on with the original plan. Safety averages the safety checks that apply to each role. It stays near 1 with cooperative partners and separates the models once the person actively intrudes (Interferer: 0.735 / 0.729 / 0.721). The Table 3 caption now lists which components Safety averages for each role, and the cross-references are fixed.

AuthorsOpenReview ↗

Kind Reminder to Reviewer FVNU

Dear Reviewer FVNU,

Thank you again for your thorough review. The discussion period is about to close, so we would like to check in briefly. We have replied to each of your points and revised the PDF accordingly (changes in blue). If our response has resolved your concerns, we would greatly appreciate a short reply or an updated assessment. If anything is still unclear, please let us know and we will answer right away.

Best regards, The Authors

Reviewer_NymCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HRC-SimBench, a simulation benchmark containing 13 human–robot collaboration tasks organized into Instructor, Collaborator, and Interferer roles. It evaluates task success alongside coordination, ordering, safety, and recovery using adapted GR00T, π0.5, and ACT policies. Generated interactions are validated through semantic auditing, comparisons with recorded human motion, and a participant study. A small real-world experiment further shows that adding simulation interaction data improves aggregate success from 3/30 to 13/30 trials.

Critical strengths
  1. The benchmark addresses interaction requirements beyond manipulation completion. Its role-based design explicitly evaluates human instruction, coordination, and interference. Table 3 shows substantial difficulties across the evaluated policies, particularly under interference. This makes the benchmark useful for studying capabilities that conventional task-success evaluation may overlook.

  2. The paper provides complementary evidence for data validity and practical utility. Main-paper Table 2 compares generated and recorded human motions, while §5.2 reports semantic auditing and a participant study. Table 4 shows improvements across all three real-world tasks. The data ablation and confidence intervals in Appendix H further support these results, although the physical evaluation remains small.

Critical weaknesses
  1. The relationship between success and order compliance requires clarification. Appendix A states that successful episodes must satisfy script-level ordering constraints, yet Appendix Table 23 reports CSR=0.700 and OC=0.667 for GR00T on I1. If both metrics use the same constraints and episodes, average OC should not fall below CSR. This ambiguity matters because §5.3 uses ordering and success measurements to interpret the policies’ interaction capabilities.

  2. The disruption recovery scores cannot be reconstructed from the stated protocol. DSR is defined as successful recoveries divided by disruptions. The protocol specifies ten episodes per task and one perturbation per episode, but Appendix Table 18 reports values such as 0.780 and 0.690. Additional repetitions or aggregation rules might explain these values, but should be specified because recovery evaluation is central to the benchmark.

Questions for authors
  1. Do CSR and OC use different constraints or event sets? Please explain the CSR=0.700 and OC=0.667 result for I1/GR00T, and the nonzero order-violation failures when OC=1.000.

  2. What numerators, denominators, and aggregation procedure produce the DSR values in Table 18, given ten evaluation episodes and one perturbation per episode?

1 comment
AuthorsOpenReview ↗

Response to Reviewer NymC

Thank you for checking our numbers so carefully. Both questions come from the metrics measuring different things, and Appendix A now makes the definitions explicit.

  1. Why is OC (0.667) lower than CSR (0.700) on I1, and why do order failures appear when OC = 1.000? CSR gives a single verdict per episode: did the robot reach the goal, in the right order, without unsafe contact? OC is an event-level checklist: of the steps the script requires, what fraction were carried out as specified? In Instructor tasks the checklist has three items: (1) do not move before the gesture appears, (2) then deliver, and (3) wait until the pointing hand has been held still for 0.5 s (Table 5; Listing 1). The third item is part of the interaction protocol but not a success condition. Picture a helper who sets off the moment you raise your arm instead of waiting until you finish pointing. They still reach the right spot, but they skipped a step.

That is what all three policies do: they start moving once the gesture is visible, before the 0.5 s hold is complete. A successful episode therefore still scores OC = 2/3. This is why OC is 0.667 across all Instructor tasks and models, and why it can sit below CSR. It is also consistent with the premature-action failures reported for Instructor tasks in Table 25.

The "Order" category in Table 25 measures something different. Table 25 assigns each failed rollout its main cause, and "Order" there means the robot's own steps were out of sequence, for example trying to place before the handover was complete. These are not the human–robot script events that OC counts. We have renamed the category "Task-step order" to make the distinction clear.

  1. How can DSR be 0.780 with ten episodes and one perturbation each?

DSR is graded, not a 0/1 count. As the caption of Table 18 notes, recovery is timed. A robot that backs off and restores the scene quickly earns nearly full credit; a slow one earns partial credit; one that never recovers is scored at the timeout, which gives zero. The average of ten graded scores need not be a multiple of 0.1. The role-level DSR in Table 24 is the mean of the three event types in Table 18. This also explains why DSR is much higher than Interferer CSR. DSR asks whether the robot backed off and restored the scene. CSR additionally requires that it then finished the task with no safety violation. Many rollouts manage the first but not the second, which is exactly the failure pattern we highlight in Section 5.5. In the revised PDF, Eq. 11 is written in its graded form, the CSR definition states that its ordering condition covers before/after relations, and Appendix A walks through the Instructor example for OC.

Reviewer_McJoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HRC-SimBench, a simulation benchmark for evaluating human–robot collaboration beyond isolated manipulation. Its 13 tasks span Instructor, Collaborator, and Interferer roles. Executable scenario scripts specify roles, temporal constraints, human-motion variants, and success conditions; interaction-specific metrics assess cue response, synchronization, protocol compliance, safety, and recovery. The authors evaluate adapted GR00T N1.5, π0.5, and ACT policies, which struggle most on Interferer tasks. In a small SO-100 study, simulation plus real-data adaptation raises physical-task success from 0.10 to 0.43, compared with real-data adaptation alone.

Critical strengths
  1. The problem is important and well motivated. Many VLA evaluations treat people as background context while scoring object-level success. By making a person's instruction, motion, or intervention part of the task, HRC-SimBench can expose a robot that reaches the right final state but acts at the wrong time, ignores a cue, or gets too close to a person. The three-role breakdown makes these failures easier to interpret.

  2. The scenario-script abstraction is a strong design choice. It specifies the interaction contract—who acts, what must happen, and when—separately from a policy's particular action sequence. Reusing the contract across object poses, human trajectories, and timing variants creates controlled comparisons, while the interaction metrics help attribute failures to grounding, synchronization, protocol, or safety rather than to an undifferentiated task-success score.

  3. The layered validation is a strength. The authors combine simulation executability checks with a manual semantic audit, retaining 786 of 2,000 candidates; independent annotation of 100 trajectories yields 94% agreement (κ = 0.87). They also compare generated motions with recorded human motion and report feedback from a 36-person validity study. These checks strengthen confidence that the retained scenarios express the intended interactions.

Critical weaknesses
  1. My main concern is the extent to which scripted human behavior tests reciprocal collaboration. The benchmark varies human trajectories, and some Interferer events are triggered by the robot's progress, but the human trajectory and perturbation schedule are selected before evaluation and replayed across policies (Appendix F). In real collaboration, people adjust their timing, path, or instructions after observing the robot. A policy could respond well to fixed interventions yet struggle with a reactive partner, or vice versa. The controlled design is useful for diagnosis, but it supports a narrower claim about response to specified human events than the paper's broader framing of collaborative intelligence. The authors should make this boundary clearer.

  2. My second concern is that the experiments do not yet separate a capability deficit from a data or adaptation deficit. Section 5.1 uses 50 adaptation trajectories and 10 held-out trajectories per task; Appendix G.1 reports a single adaptation seed. The common data split helps, but GR00T N1.5 and π0.5 use LoRA while ACT uses full fine-tuning, with different training budgets (Appendix G.2). Thus the 0.100 Interferer success rate across all three policies (Table 3) does not by itself establish an intrinsic lack of collaborative capability. The gains from interaction-structured data in Table 27 and from Sim+Real training in Table 4 suggest that data distribution matters, although the latter also changes total data volume. Data-scaling experiments with multiple seeds would clarify how much of the gap persists.

Questions for authors
  1. How would model rankings and failure modes change if the human responded to the robot's pauses, approach, or errors?

  2. How do Interferer success, safety, and recovery change as adaptation data grows beyond 50 trajectories per task?

3 comments
AuthorsOpenReview ↗

Response to Reviewer McJo

Thank you for the positive and thoughtful review.

  1. Scripted vs. reactive humans. Thank you for the suggestion; we have adopted your framing. HRC-SimBench measures how a policy responds to specified human events, which is narrower than full two-way collaboration; the introduction and a new Limitations paragraph after the Conclusion now say so. Replaying the same human for every policy is what keeps the comparison fair; with a reactive partner, the outcome would depend on the partner model as much as on the robot. We expect a reactive partner to make the current failures more visible, not to hide them. The dominant Interferer failure is pressing on with the original plan (Table 25), and a person who keeps reaching in when the robot does not yield only raises the cost of that mistake. A reactive-human track is the first item in our future work.

  2. Capability or data?Data clearly matters, and Table 27 shows it. With the same robot and the same amount of data, changing only what the data teaches (object-only vs. interaction) raises Collaborator CSR from 0.133 to 0.500. Yet under the same 50-trajectory budget, Interferer stays at 0.100 while Collaborator reaches 0.500 (35/70 vs. 3/30, Fisher p ≈ 1×10⁻⁴). Recovering from interference is the hardest part for current policies under standard adaptation budgets; we do not claim it is an intrinsic limit. On the training setup: ACT has no large pretrained backbone, so full fine-tuning is its standard recipe, while the two VLAs share identical LoRA rank and alpha. In Table 4, both conditions use the same 20 real demonstrations per task (Sec. 5.4), so the gain comes from adding HRC-SimBench data.

Thank you for the revisions. The explicit distinction between scripted-event response and two-way collaboration addresses my scope concern. The matched-data ablation and clarification that you do not claim an intrinsic capability limit also help. My remaining concern is that Interferer performance is evaluated at one adaptation budget and one seed (Appendix G.1); please state this limit alongside the main results.

AuthorsOpenReview ↗

Response to Reviewer McJo's Follow-up

Thank you for the follow-up. We agree that this should be stated next to the main results, and the revised PDF now does so.

(1) Seed. The main table uses one seed per model. To check this, we retrained GR00T N1.5 with three seeds on one task per role, still with 50 trajectories per task. Each seed is evaluated on 10 held-out episodes, so each task has 30 rollouts in total:

I1 (Instructor)C1 (Collaborator)N1 (Interferer)
Successes19/3019/304/30
CSR (mean ± std over seeds)0.63 ± 0.060.63 ± 0.060.13 ± 0.06

The Interferer task is still far behind the other two, so the ordering is not a seed effect.

(2) Budget. We tested one budget, and we think it is the right one to report, for three reasons.

First, 50 trajectories per task is the standard setting for this kind of evaluation (LIBERO, for example, ships 50 demonstrations per task), so our numbers show what current policies do under normal adaptation.

Second, the budget is the same for all roles. With the same 50 trajectories, GR00T N1.5 reaches 0.500 on Collaborator tasks and 0.100 on Interferer tasks. The gap is between roles, not between budgets.

Third, these tasks differ from plain manipulation in what the policy has to generalize over. In a manipulation task the test variation is in object pose and scene, and more demonstrations cover it better. Here the test variation is in the human: when they move, how fast, and when they step in. The human's behavior changes what the policy must do while the episode is running. Our failures are of exactly this kind. The policies do not stall (Interferer IR is 0.04–0.06); they keep executing the original plan after the person intervenes (Table 25). Table 27 shows the same thing from the data side: with the same amount of data, changing what the data contains moves Collaborator CSR from 0.133 to 0.500. This suggests that the bottleneck is reacting to the person more than data coverage, so simply adding more trajectories is unlikely to make the policy adapt much better to the held-out human variants.

We have not tested larger budgets, so we leave data scaling to future work and say so in the Limitations paragraph.

Changes in the PDF (in blue). Sec. 5.3 and the caption of Table 3 now say that the results use one budget and one seed. The Limitations paragraph says the same and lists data scaling as future work. The three-seed results are in Appendix H.6 (Table 29).

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their careful and constructive feedback. All three agree that the problem is important: most VLA benchmarks leave the human out of the loop. They also find that role-conditioned scenario scripts make collaborative failures easier to interpret. Below are the main concerns and what we changed.

  1. What the benchmark claims. The simulated human follows a script and does not react to the robot. We now state our claim precisely: HRC-SimBench measures how a policy responds to specified human cues, partner motion, and interventions, under identical conditions for every policy. A new Limitations paragraph after the Conclusion names reactive partners as the next step.
  2. Collaboration gap vs. manipulation gap. Three results in the paper separate the two. (i) Training on manipulation-only data leaves the failures in place, while interaction data reduces them (Table 27, now summarized in Sec. 5.3). (ii) Policies execute the nominal steps correctly (OC = 1.000 on Collaborator and Interferer tasks) and fail only when the human intervenes. (iii) The recorded failure causes are interactive: acting too early, lagging the partner, and failing to yield or recover (Table 25).
  3. Metric definitions. Appendix A now explains how OC and DSR are computed and why they can differ from CSR.
  4. Presentation and statistics.We corrected the description of the evaluation split (650 training / 130 held-out episodes), added significance tests (Sec. 5.3), added the Idle Ratio to Table 3, cited Hoffman's fluency metrics, extended the related-work comparison (Sec. 2 and Table 1), and fixed cross-references. All changes are marked in blue in the revised PDF.
AuthorsOpenReview ↗

Kind Reminder from the Authors

Dear Reviewers,

Thank you again for your time and for the careful review. We have posted our response and uploaded a revised PDF with all changes marked in blue. As the discussion period is ending soon, we would be grateful if you could let us know whether our response addresses your concerns, or if anything remains unclear. We are happy to provide further clarification before the deadline.

Best regards,

The Authors

AuthorsOpenReview ↗

Appreciation to the Reviewer

Thank you for re-evaluating our manuscript and raising the score. The constructive suggestions from the previous round have been very helpful in improving the quality of this work!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.