acceptodds
Under review as a conference paper at ICLR 2027

BodyBound: A Benchmark for Diagnosing Robot-Policy Responses to Body Changes Beyond Task Success

Abstract

Task success does not by itself show whether a robot policy adapts when its body changes. A policy may fail before the intervention, or the same action sequence may work on both bodies. BodyBound is a controlled benchmark for separating these cases. It pairs executions with the same scene, goal, initial state, and rollout budget while changing one body property: joint limits, jaw capacity, finger-force limit, or wrist-guard geometry within a Panda topology. An independent trace checker is combined with four diagnostics: familiar-start competence, expert attainability witnesses, reciprocal body-description interventions, and fixed-action replay. We release 100 simulated RGB-D demonstrations and evaluate HPT and Diffusion Policy on 60 fresh paired scenes (720 matching-description rollouts). On the public test split, success is 4.2% for HPT and 19.4% for Diffusion Policy; all six checkpoints obtain zero successes in Scene OOD. The learned policies also fail on many familiar starts, limiting attribution of failures to body changes. In a competent Portal reference, all 16 fresh pairs succeed with zero description benefit and shared successful actions; a Grasp reference shows a positive description benefit despite shared solutions. BodyBound therefore measures competence, physical response, and evidence of adaptation separately.

Primary area:
datasets and benchmarks
Keywords:
Embodied Intelligence, Robot Adaptation, Benchmarking, Embodied Planning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WAVTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BodyBound, a benchmark for diagnosing robot-policy adaptation to body changes beyond task success. It evaluates policies under controlled physical interventions using familiar-start competence, expert attainability witnesses, reciprocal body-description interventions, and fixed-action replay. Experiments with HPT and Diffusion Policy reveal limited task competence and demonstrate the ambiguity of interpreting adaptation from success alone.

Critical strengths
  1. The motivation is good. The paper identifies an important limitation of task-success metrics, success does not necessarily imply adaptation, and failure cannot always be attributed to body changes.

  2. The design of the method makes sense. The paired interventions, reciprocal body-description tests, and fixed-action replay provide complementary diagnostics for understanding policy behavior.

  3. The experiments report both standard deviations and resampling intervals, including the mean and sample standard deviation across three training seeds. This strengthens the credibility of the experimental results.

Critical weaknesses
  1. The paper lacks more advanced baseline methods. HPT and Diffusion Policy achieve only 4.2% and 19.4% success on the public test split, respectively, and also fail on many familiar training starts.

  2. Varying the Panda robot’s body parameters in simulation is a meaningful experiment. However, these experiments alone do not fully establish the effectiveness of the proposed benchmark. I suggest extending the evaluation to common changes in control settings, such as control frequency and control method, and exploring different actuators and robot arms.

  3. The lack of real-world robot experiments/benchmarks is my main concern and a key factor in my rating. Simulation alone cannot fully establish the effectiveness of the benchmark in real-world settings.

  4. Although the authors conduct extensive simulation experiments, Figure 1 still uses VLM-generated robot-arm images. I suggest replacing them with images from the actual simulator. This is a presentation suggestion and does not affect my rating.

Questions for authors

What I would most like to see is a complete example in which both body configurations can complete the task, but the original action sequence cannot be directly reused. One learned policy should successfully adjust its behavior to the body change while another fails to do so, and BodyBound’s diagnostics should clearly distinguish the two. Does the current benchmark contain such an example?

Reviewer_gSmXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper manuscript introduces BodyBound, a benchmark for testing how robot policies respond to changes in their robot bodies. BodyBound changes properties such as joint limits, gripper opening, and gripping force while keeping the scene and task fixed to evaluate how physical properties of the robot itself will affect the policy performance. It uses additional checks to understand whether success requires adjusting to the changed body or whether the same actions work for both bodies. Experiments with HPT and Diffusion Policy show frequent failures even on familiar tasks, making their ability to adapt difficult to assess.

Critical strengths
  1. The paper introduces an important evaluation problem: task success alone does not show whether a robot policy adapts to changes in the robot body. This is a useful direction for understanding robot behavior beyond overall success rates.
  2. The benchmark uses controlled comparisons that keep the scene and task fixed while changing a robot body property. Further additional checks, such as testing familiar starts, swapping body descriptions, and replaying the same actions on both bodies, help interpret the results by setting up controlled baselines.
Critical weaknesses
  1. The learned baselines both HPT and Diffusion Policy have low success even on familiar training starts. This makes it difficult to tell whether failures are caused by body changes or by weak basic task performance. Stronger learned baselines would make the evaluation of adaptation more convincing.
  2. The training dataset is small and uneven, with only 72 training demonstrations. The evaluation also uses only two scenes per task family per evaluation condition. More balanced training data and evaluation on more scenes would better support the benchmark’s usefulness.
  3. Task feasibility remains unclear for many evaluation cases. The expert succeeds on only 57 of 120 instances. Although expert failure does not prove that the remaining instances are impossible, it makes policy failures harder to interpret. The authors provide some analysis of expert failures, but a more detailed breakdown and additional successful reference executions would help clarify task feasibility.
  4. The experiments use only one simulated robot arm. Tests on real robots or different robot designs would help show whether the benchmark is useful beyond this setup.

Overall, the weak performance even on familiar tasks is my main concern. The authors acknowledge this limitation, but the current experiments provide limited evidence of how useful the benchmark is for evaluating adaptation in competent learned policies.

Questions for authors
  1. What causes the low success rates even on familiar tasks? Does performance improve with more demonstrations or additional training?
  2. Can you repeat the body-change experiments with a learned policy that reliably solves the tasks before the body change? This would help separate basic task failures from difficulties adapting to the changed body.
Reviewer_YXWBOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is providing a benchmark for diagnosing how manipulation policies respond to body changes. It uses paired interventions: within a fixed scene, two Panda variants differ in one body parameter (joint range, jaw opening, finger force limit, or guard size), with goal, initial state, and budget held constant. Six task families (Reach, Grasp, Force, Portal, Spool, Coupling) are evaluated by an independent checker on measured traces. The framework separates four questions: competence, physical response, description benefit, and action necessity.

Two policies (HPT, Diffusion Policy) are trained on 100 demos across 60 fresh paired scenes and five conditions. Key findings: success is low (HPT 4.2%, DP 19.4%); both fail on familiar starts; matching-description benefit is near zero (C≈0); DP shows physical response (P₀=+6.1, P₁=+4.4) but no description benefit; and 36/110 expert sequences succeed unchanged on both bodies. Limitations include one topology, two scenes per family, and fixed training budgets.

Critical strengths

Clear problem framing. Separating "competence," "physical response," "description benefit," and "action necessity" is a useful methodological contribution. Controlled paired interventions. Changing one body parameter at a time while fixing scene, goal, initial state, and budget is methodologically sound. Detailed documentation. Task predicates, simulator settings, body ranges, and reset corrections are well documented. Honest limitations. Single topology, imbalanced demos, two scenes per condition, and fixed training budgets are explicitly acknowledged.

Critical weaknesses
  • Central claim confounded by floor effects. The "C≈0 means descriptions are ignored" claim is not causally established. HPT success is 1.1–3.9% across all four cells, leaving no room to detect a description effect. Zero-feature controls are similarly explained by floor effects. The paper acknowledges this possibility but does not resolve it.
  • Statistical basis is too weak. 24 members per condition, 4 per family, 3 seeds repeating the same scenes (not increasing diversity), n=3 SDs explicitly not confidence intervals, and bootstrap intervals conditional on two scenes per stratum. The numbers are not stable enough for the claims.
  • Limited benchmark scope. Only two learned policies (HPT, DP), four fixed body pairs, one Panda topology, and analytic references on only two tasks. This is not a robustness benchmark; it cannot support policy comparison or leaderboard use.
  • No diagnosis-to-improvement loop. The framework is proposed but never validated: no policy is improved based on the diagnosis, and the four categories are not shown to lead to different remedies or to be exhaustive/independent.
  • No failure-mode analysis. Despite being a "failure attribution" benchmark, only binary success/failure is reported. Stage-wise pass rates (G→H→P, C→P) and physical failure mechanisms are not systematically analyzed.
  • Weak baselines constrain generality. 50 epochs, frozen transformer, byte-token encoders without pretrained language knowledge, RGB-only (no depth), 100 demos (zero for Spool/Coupling), and poor familiar-start competence (HPT 1/8, DP 2–4/8). The conclusion that "policies cannot use body descriptions" may reflect training budget and encoder weaknesses rather than a general property of policies.
  • Rhetorical overreach. The body text is cautious ("descriptive," "not evidence that they ignore the body"), but the title, abstract, and conclusion send a stronger message than the evidence supports.
Questions for authors

How do you rule out floor effects in the C≈0 claim? Are the reported contrasts statistically valid given the sample size? How can these diagnostics be used to improve a policy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.