- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BARM, aiming to serve as a router to distinguish OOD situations in robot manipulation that need VLM or human help.
Critical strengths
The paper aims to solve a problem which is important nowadays as agent-based robot manipulation is a trend, which makes simpler ways to distinguish OOD cases during rollout crucial. The method in this paper is lightweight.
Critical weaknesses
- The experiments are fully conducted in simulated environments, while real-world environments are lacked. Real-world scenes are more diverse and uncontrolled.
- The method relies on SNGP to detect OOD cases where VLM or human helps are needed. However, in broader settings and open world with various scenes and different tasks, SNGP might be unreliable for such detection In the paper, the authors use only hundreds of samples for training and testing, which is a very small one compared pretraining-scale or multitask datasets used in actual practice. Thus, while the authors aim to solve a practical problem, the method itself might be not practical enough.
Update Rating increased.
Questions for authors
I suggest the authors should conduct experiments under broader settings and with a much larger dataset and see if the performance will degrade, which is probable.
2 comments
Follow-up to eYYB (W1): Real-Robot Evaluation
We thank the reviewer again for suggesting a real-world evaluation. As promised in our response to W1, we report the results on a real robot. The table reports net reward, the task reward minus the cost of the selected mode, as mean ± standard deviation over five seeds.
Setup. We deploy BARM on a real myPalletizer 260 arm with a single third-person camera. The three execution modes are those of Section 5.1, with in place of the diffusion policy. We fine-tune only on demonstrations of simple tasks collected on this robot. Simple tasks ask the arm to reach one object on the table, and composite tasks ask it to reach multiple objects in a given order. We collect 244 real-robot rollouts spanning simple and composite tasks. The router is trained offline on simple tasks only, and online evaluation uses the same costs as on CALVIN and the same number of episodes, task mixture, seeds, and 22 baselines as in Section 5.1.
Results.
| Method | Real robot |
|---|---|
| Oracle | 0.923 ± 0.002 |
| BARM | 0.857 ± 0.010 |
| Bandit methods | |
| Neural-LinUCB | 0.840 ± 0.009 |
| NeuralTS | 0.735 ± 0.005 |
| EE-Net | 0.684 ± 0.050 |
| Detection methods | |
| SNGP+Conf | 0.746 ± 0.012 |
| Sentinel | 0.743 ± 0.011 |
| FIPER | 0.740 ± 0.015 |
| DeeR-VLA | 0.718 ± 0.009 |
| KnowNo | 0.711 ± 0.014 |
| CogVLA | 0.651 ± 0.028 |
| IntroPlan | 0.648 ± 0.007 |
| RL & learned routing | |
| DQN | 0.790 ± 0.013 |
| LLM Routing | 0.736 ± 0.054 |
| MCDrop | 0.726 ± 0.040 |
| PPO | 0.701 ± 0.011 |
| REINFORCE | 0.688 ± 0.006 |
| FiS-VLA | 0.646 ± 0.018 |
| VADER | 0.644 ± 0.007 |
| AutoVLA | 0.548 ± 0.018 |
| Fixed policies | |
| Always Arm2 | 0.770 ± 0.013 |
| Random | 0.682 ± 0.012 |
| Always Arm3 | 0.648 ± 0.004 |
| Always Arm1 | 0.544 ± 0.018 |
With the execution modes running on a physical robot, BARM achieves the highest net reward, outperforming all 22 baselines.
We will add these results to the main text of the revised paper, with the full setup and photos of the platform in the appendix and a demo video in the supplementary material.
Thank You for Raising the Score to 4
We thank the reviewer for reading our response and for raising the score to clear acceptance.
We will incorporate the results and clarifications from our response into the revised paper.