acceptodds
Under review as a conference paper at ICLR 2027

VIA: Visual Interface Agent for Robot Control

Abstract

Robot manipulation is a complex task that requires visual perception, physical reasoning, planning, and closed-loop control. General-purpose foundation models (FMs) have grown remarkably capable of some of these, especially perception and reasoning. Inspired by the growing ability of FM-powered agents to operate software through visual interfaces, we ask whether that same competence suffices to control a robot directly given the right interface. We present VIA (Visual Interface Agent), a framework that recasts robot control as an agentic task where an off-the-shelf agent drives a manipulator directly through an interface. The interface is a virtual workspace of interactable visual components where the agent can probe pixels of interest with mouse-like tools and command a virtual end-effector to set new target poses with a small set of general tools. We show that VIA enables agents of varying capabilities to perform diverse tabletop manipulation tasks in simulation. We also use VIA to drive an off-the-shelf mobile manipulation robot in the real world to complete tasks that require both navigation and manipulation. Both settings are zero-shot, requiring no robot training or additional action primitives. Performance scales well with the size and strength of the underlying FMs. These results suggest that frontier agents already possess skills that transfer directly to robot control given the right interface: your coding or computer-use agent is, in a sense, secretly a robot-control agent.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics, manipulation, agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NvNMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new methodology to use foundation models to solve robot manipulation tasks. It reformulates manipulation as an agentic task. It builds interfaces to create an interactive visual workspace, where foundation models can interact with, probe areas of interest, and do some virtual trial and error.

The paper evaluates by using state-of-the-art foundation models to solve manipulation tasks in both simulation and real world under this framework. State-of-the-art foundation models can achieve 100% success rate on some tasks zero-shot. They also show that, without the interface introduced in this paper, success rate drops significantly for most foundation models.

Critical strengths
  • The methodology of using foundation models to solve robotics manipulation problems is novel. It is different from all previous usage, including VLAs, hierarchical control, visual prompting, and code-as-policies.

  • The designed interface improves the success rate of most foundation models.

  • Experiment results are valuable to the community. It shows that existing foundation models are already capable of solving some manipulation problems zero-shot, via a good interface.

Critical weaknesses
  • One major limitation of this work is that it does not compare VIA with strong baselines. The only baseline it compares with is a direct use of foundation models without any interface. It is unclear how VIA compares with other paradigms of using foundation models, more specifically, the 4 categories mentioned in Related Works: VLAs, hierarchical control, visual prompting, and code-as-policies. Moreover, there is no baseline for real world evaluation (Table 3). It is unclear how superior VIA is compared with other methods in the real world.

  • There is no discussion of inference speed or latency in the paper, while the only thing being discussed is number of tool calls (Appendix B). Inference speed of foundation models is slow. Even worse, an important part of VIA is mini-loops that allow foundation models doing trial-and-error. This means more inference calls to foundation models and slower speed. All these factors could bring the huge challenge of using VIA to solve robotics tasks that require fast response.

Questions for authors
  • How does the performance of VIA compare with other paradigms of using foundation models to solve robotics tasks, including VLAs, hierarchical control, visual prompting, and code-as-policies? Could authors report baseline results for real world evaluation?

  • Could authors report inference time of VIA, and compare that with inference time of other robotics methods using foundation models? Do authors have any ideas on how to improve inference speed of VIA?

1 comment

Thank you for your thoughtful feedbacks. We would like to make the following response:

How does the performance of VIA compare with other paradigms of using foundation models to solve robotics tasks, including VLAs, hierarchical control, visual prompting, and code-as-policies? Could authors report baseline results for real world evaluation?

Thanks for the suggestion. We evaluated pi0.5, a vision-language-action model, and CaP-X, a strong code-as-policy method, on the same six simulation tasks. The table reports success rates over 10 trials per task.

Taskpi0.5CaP-X (Sol 5.6)VIA (Sol 5.6)
Turn on stove100%90%100%
Open drawer10%20%80%
Put bowl on plate90%90%100%
Square0%20%30%
Rainbow0%20%100%
T-block0%50%30%
Average33.3%48.3%73.3%

The pi0.5 column uses the released pi05_libero checkpoint, which was fine-tuned on the libero tasks. We do not perform any additional fine-tuning, because all other methods do not use any task specific fine-tuning. Our CaP-X adaptation uses Codex with Sol 5.6 for code generation with its existing perception and action primitives while their original paper used raw Gemini models without coding agent harness. So our CaP-X implementation is substantially modernized and should perform better than the original. VIA performs substantially better than both methods.

One reason for not comparing against the methods mentioned in the related work is how different these methods are in nature. They are trained on vastly different datasets (VLA on massive robot motion specific data), requires specialized primitives (CaP, visual prompt etc). During majority period of the time when this project was developed, to the best of our knowledge, there were no major results showing that agents are capable at controlling robots directly without all these special training nor robot specific primitives like GraspNet or Molmo point. This is also demonstrated partially by the substantial improvement that VIA brings to the pre-Astra models. The Astra model was only released less than a month before the submission time of this paper, which would have been considered as concurrent in traditional conference publication cycle.

The focus of the real world evaluation is to show that the system can be applied to off-the-shelf mobile robot with noisy perceptions in room-scale to complete a diverse range of tasks. Some tasks we do here would normally require hundreds of demonstrations to fine-tune a VLA policy to perform. Our simulation ablations have clearly demonstrated the benefits of VIA. Running the ablation again in real world evaluations would be a time-consuming effort for very little extra information.

Could authors report inference time of VIA, and compare that with inference time of other robotics methods using foundation models? Do authors have any ideas on how to improve inference speed of VIA?

The inference speed of VIA as a major limitation, which we have discussed openly in the paper. Currently, it is orders of magnitudes slower than other methods. We do not claim any applicability of the proposed method on tasks beyond quasi-static tasks. For the real world evaluations that we run sequentially, most episodes finished within around 10 mins, and roughly 30% of the time was spent on robot execution and 70% of the time was spent on model inference.

There are natural ways to make this system faster:

  • As model improves, these capabilities will emerge or get distilled into smaller models. There are already evidence from new work that came out recently on finetuning a 27B model to perform tasks [1].
  • Making the reasoning and execution async.

[1] Training Reasoning Models in Simulation to Natively Use Real Robots, blog: https://mit-han-lab.github.io/pear-blog/

Reviewer_yKnpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VIA, a framework that enables an off-the-shelf visual agent stage and refines end-effector targets in a virtual workspace before executing them. The paper evaluates six simulation tasks and five real-world mobile manipulation tasks without robot-specific fine-tuning.

Critical strengths
  1. The interface offers a practical way to check and revise a target before moving the robot. The combination of simulation and real-world results makes the system worth studying.

  2. The tasks cover articulated objects, precise placement, long-horizon assembly, and deformable objects. The tool and prompt descriptions are helpful.

Critical weaknesses
  1. The baseline changes several things at once: VIA adds depth probing, pixel-to-pose mapping, local motion tools, camera control, and target preview. The current comparison does not isolate the benefit of preview and refinement.

  2. The evaluation needs tightening. Table 7 counts an unrun Opus T-block baseline as 0%, and some Rainbow grades are provisional. The real-world study has only five trials per task and no baseline; costs exclude failed episodes. These gaps make the reliability and efficiency claims hard to assess.

Questions for authors
  1. With the same depth-probing and motion tools on both sides, how much does target preview improve success?

  2. Can you complete the missing baseline runs, finalize Rainbow grading, and report costs including failures?

  3. Does T-block performance hold when the default prompt no longer hints that the blue base needs reorientation?

1 comment

Thank you for your thoughtful feedback. We would like to make the following response:

The baseline changes several things at once... With the same depth-probing and motion tools on both sides, how much does target preview improve success?

We ran a version of VIA (Sol 5.6) with only the depth probing tool (hover), without any mini-loop that allows the preview and refinement of the waypoints. The performance drops from 73.3% to 56.7%. So the target preview and the ability to refine them in the mini-loop is helpful, especially for precision demanding tasks like "Bowl on plate" and "T-block", as well as operating articulated object such as "Open drawer".

Here is the detailed comparison per task.

TaskVIA hover (Sol 5.6)VIA full (Sol 5.6)
Turn on stove9/1010/10
Open drawer2/108/10
Bowl on plate7/1010/10
Square7/103/10
Rainbow9/1010/10
T-block0/103/10
Overall34/60 (56.7%)44/60 (73.3%)

Table 7 counts an unrun Opus T-block baseline as 0%, and some Rainbow grades are provisional. Can you complete the missing baseline runs, finalize Rainbow grading, and report costs including failures?

Thanks for catching this. It is totally an oversight on our side. The numbers have been finalized before submitting but we forgot to update that table in the Appendix. The final aggregated numbers in Fig 3 use the complete results. The numbers are accurate and do not need to be changed. We have updated the Appendix to remove the outdated comments.

The real-world study has only five trials per task and no baseline.

The real world study is mainly as a qualitative evaluation to show that the method works on an off-the-shelf, commercially available robot. It also extend the applicability of the method to a mobile robot in a cluttered, room scale environment. The ablations have been conducted in the simulation tasks, so our real evaluations are focused on the generalization and task diversity aspects.

Costs exclude failed episodes.

Cost including the failure episodes is not informative at comparing the true efficiency between models because it couples efficiency and capability, and it is heavily skewed by how long we allow them to run. Nonetheless, here is the table with cost including the failure episodes for the GPT models in "()" for reference.

TaskTerra 5.6Sol 5.6Astra 6
Turn on stove\1.1(1.1 (\\1.3)\2.5(2.5 (\\2.5)\1.0(1.0 (\\1.0)
Open drawer\2.8(2.8 (\\2.8)\1.0(1.0 (\\1.0)\1.3(1.3 (\\1.3)
Put bowl on plate\1.9(1.9 (\\2.2)\1.7(1.7 (\\1.7)\2.5(2.5 (\\2.5)
Square-- (\$5.0)\3.5(3.5 (\\8.9)\2.8(2.8 (\\2.8)
Rainbow\1.6(1.6 (\\2.7)\4.5(4.5 (\\4.5)\7.6(7.6 (\\7.6)
T-block\4.9(4.9 (\\7.9)\3.8(3.8 (\\12.1)\3.7(3.7 (\\3.7)
Overall\2.4 (\3.7)**\2.8(2.8 (\\5.1)**\3.2(3.2 (\\3.2)

For example if we look at the cost of Sol vs Astra on the Square task, the Astra's cost is low even on the success only episodes, showing true efficiency improvement. If we instead compare \2.8vsthefailure−includedcostof2.8 vs the failure-included cost of \\8.9, we are not sure if Astra is more efficiency purely because it succeeds more.

Does T-block performance hold when the default prompt no longer hints that the blue base needs reorientation?

Good question. Yes, in fact all models are able to mentally solve the puzzle part that the blue base needs reorientation even without the hint. In the default prompt, we do not reveal how the reorientation should be done, so the models still need to figure that out. The key difference between models is in the precision of execution.

Reviewer_48iYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VIA lets general-purpose agents control a robot through a visual workspace. Agents adjust and inspect a virtual gripper before executing waypoints. The paper evaluates six simulation tasks and five real-world mobile-manipulation tasks without robot-specific fine-tuning.

Critical strengths

Staging a waypoint visually before execution is a sensible interface design. The mobile-manipulation results are promising and demonstrate that the approach can work beyond clean simulation observations.

Critical weaknesses
  1. Some baseline results are incomplete. Table 7 counts an unexecuted Opus 4.8 T-block evaluation as 0%, and the Fable 5 Rainbow result is provisional. These entries feed into the reported averages. An unrun experiment should not be treated as an observed failure, particularly when the comparison is central to the paper.
  2. The comparison does not isolate which interface feature helps: the baseline simultaneously loses probing tools and virtual waypoint refinement. There is also no comparison with an established iterative visual-prompting approach. With Astra 6, the baseline already achieves 93.3% versus VIA's 100%, and the 25 real-world trials have no baseline. The strongest-model results therefore leave the interface's incremental benefit uncertain.
Questions for authors
  1. What are the corrected comparisons after completing the missing run and finalizing the provisional grades?
  2. At a matched inference budget, how much do probing and waypoint preview contribute separately, and does the Astra baseline remain competitive on the real robot?
1 comment

Thank you for your thoughtful feedback. We would like to make the following response:

Some baseline results are incomplete. What are the corrected comparisons after completing the missing run and finalizing the provisional grades?

The outdated comments in the Appendix is an oversight of us. We have indeed finished the evaluation when we produce the Fig 5 in the main text but forgot to update the draft Table 8 in the Appendix. Everything in the main text was up-to-date and the numbers in Table 8 remains unchanged after the completed evaluation. Thanks again for catching this and our updated manuscript has reflected the updates.

The comparison does not isolate which interface feature helps... At a matched inference budget, how much do probing and waypoint preview contribute separately, and does the Astra baseline remain competitive on the real robot?

As part of the rebuttal, we ran a version of VIA (Sol 5.6) with only the depth probing tool (hover), without any mini-loop that allows the preview and refinement of the waypoints. The performance drops from 73.3% to 56.7%. Here is the detailed comparison per task.

TaskVIA hover (Sol 5.6)VIA full (Sol 5.6)
Turn on stove9/1010/10
Open drawer2/108/10
Bowl on plate7/1010/10
Square7/103/10
Rainbow9/1010/10
T-block0/103/10
Overall34/60 (56.7%)44/60 (73.3%)

We hope these updates help address both of the core concerns raised in the review. Thanks again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.