- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BodyBound, a benchmark for diagnosing robot-policy adaptation to body changes beyond task success. It evaluates policies under controlled physical interventions using familiar-start competence, expert attainability witnesses, reciprocal body-description interventions, and fixed-action replay. Experiments with HPT and Diffusion Policy reveal limited task competence and demonstrate the ambiguity of interpreting adaptation from success alone.
Critical strengths
-
The motivation is good. The paper identifies an important limitation of task-success metrics, success does not necessarily imply adaptation, and failure cannot always be attributed to body changes.
-
The design of the method makes sense. The paired interventions, reciprocal body-description tests, and fixed-action replay provide complementary diagnostics for understanding policy behavior.
-
The experiments report both standard deviations and resampling intervals, including the mean and sample standard deviation across three training seeds. This strengthens the credibility of the experimental results.
Critical weaknesses
-
The paper lacks more advanced baseline methods. HPT and Diffusion Policy achieve only 4.2% and 19.4% success on the public test split, respectively, and also fail on many familiar training starts.
-
Varying the Panda robot’s body parameters in simulation is a meaningful experiment. However, these experiments alone do not fully establish the effectiveness of the proposed benchmark. I suggest extending the evaluation to common changes in control settings, such as control frequency and control method, and exploring different actuators and robot arms.
-
The lack of real-world robot experiments/benchmarks is my main concern and a key factor in my rating. Simulation alone cannot fully establish the effectiveness of the benchmark in real-world settings.
-
Although the authors conduct extensive simulation experiments, Figure 1 still uses VLM-generated robot-arm images. I suggest replacing them with images from the actual simulator. This is a presentation suggestion and does not affect my rating.
Questions for authors
What I would most like to see is a complete example in which both body configurations can complete the task, but the original action sequence cannot be directly reused. One learned policy should successfully adjust its behavior to the body change while another fails to do so, and BodyBound’s diagnostics should clearly distinguish the two. Does the current benchmark contain such an example?