- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a new methodology to use foundation models to solve robot manipulation tasks. It reformulates manipulation as an agentic task. It builds interfaces to create an interactive visual workspace, where foundation models can interact with, probe areas of interest, and do some virtual trial and error.
The paper evaluates by using state-of-the-art foundation models to solve manipulation tasks in both simulation and real world under this framework. State-of-the-art foundation models can achieve 100% success rate on some tasks zero-shot. They also show that, without the interface introduced in this paper, success rate drops significantly for most foundation models.
Critical strengths
-
The methodology of using foundation models to solve robotics manipulation problems is novel. It is different from all previous usage, including VLAs, hierarchical control, visual prompting, and code-as-policies.
-
The designed interface improves the success rate of most foundation models.
-
Experiment results are valuable to the community. It shows that existing foundation models are already capable of solving some manipulation problems zero-shot, via a good interface.
Critical weaknesses
-
One major limitation of this work is that it does not compare VIA with strong baselines. The only baseline it compares with is a direct use of foundation models without any interface. It is unclear how VIA compares with other paradigms of using foundation models, more specifically, the 4 categories mentioned in Related Works: VLAs, hierarchical control, visual prompting, and code-as-policies. Moreover, there is no baseline for real world evaluation (Table 3). It is unclear how superior VIA is compared with other methods in the real world.
-
There is no discussion of inference speed or latency in the paper, while the only thing being discussed is number of tool calls (Appendix B). Inference speed of foundation models is slow. Even worse, an important part of VIA is mini-loops that allow foundation models doing trial-and-error. This means more inference calls to foundation models and slower speed. All these factors could bring the huge challenge of using VIA to solve robotics tasks that require fast response.
Questions for authors
-
How does the performance of VIA compare with other paradigms of using foundation models to solve robotics tasks, including VLAs, hierarchical control, visual prompting, and code-as-policies? Could authors report baseline results for real world evaluation?
-
Could authors report inference time of VIA, and compare that with inference time of other robotics methods using foundation models? Do authors have any ideas on how to improve inference speed of VIA?
Rebuttal
Thank you for your thoughtful feedbacks. We would like to make the following response:
Thanks for the suggestion. We evaluated pi0.5, a vision-language-action model, and CaP-X, a strong code-as-policy method, on the same six simulation tasks. The table reports success rates over 10 trials per task.
The pi0.5 column uses the released
pi05_liberocheckpoint, which was fine-tuned on the libero tasks. We do not perform any additional fine-tuning, because all other methods do not use any task specific fine-tuning. Our CaP-X adaptation uses Codex with Sol 5.6 for code generation with its existing perception and action primitives while their original paper used raw Gemini models without coding agent harness. So our CaP-X implementation is substantially modernized and should perform better than the original. VIA performs substantially better than both methods.One reason for not comparing against the methods mentioned in the related work is how different these methods are in nature. They are trained on vastly different datasets (VLA on massive robot motion specific data), requires specialized primitives (CaP, visual prompt etc). During majority period of the time when this project was developed, to the best of our knowledge, there were no major results showing that agents are capable at controlling robots directly without all these special training nor robot specific primitives like GraspNet or Molmo point. This is also demonstrated partially by the substantial improvement that VIA brings to the pre-Astra models. The Astra model was only released less than a month before the submission time of this paper, which would have been considered as concurrent in traditional conference publication cycle.
The focus of the real world evaluation is to show that the system can be applied to off-the-shelf mobile robot with noisy perceptions in room-scale to complete a diverse range of tasks. Some tasks we do here would normally require hundreds of demonstrations to fine-tune a VLA policy to perform. Our simulation ablations have clearly demonstrated the benefits of VIA. Running the ablation again in real world evaluations would be a time-consuming effort for very little extra information.
The inference speed of VIA as a major limitation, which we have discussed openly in the paper. Currently, it is orders of magnitudes slower than other methods. We do not claim any applicability of the proposed method on tasks beyond quasi-static tasks. For the real world evaluations that we run sequentially, most episodes finished within around 10 mins, and roughly 30% of the time was spent on robot execution and 70% of the time was spent on model inference.
There are natural ways to make this system faster:
[1] Training Reasoning Models in Simulation to Natively Use Real Robots, blog: https://mit-han-lab.github.io/pear-blog/