acceptodds
Under review as a conference paper at ICLR 2027

VLMs Play Robotics: Can VLMs Control Locomotion and Manipulation?

Abstract

Vision-language models (VLMs) can reason over images and write code, but whether these abilities transfer to robotic control is unclear. To answer this question, we define several interfaces—interaction patterns—that correspond to different resources that VLMs commonly have access to during deployment. We then evaluate frontier and open-weight models on those interfaces across toy tasks, quadrupeds, humanoids, and robotic arms. We find that models are stronger at higher-level control, although nearly every capability we test shows generational performance gains within each model family. Specific interface and tool choices change performance by an order of magnitude, meaning narrow evaluations may under- or overestimate a model’s ability to exert control. Though overall success rates are low and other factors such as latency prevent effective deployment, our experiments indicate that generalist VLMs possess nascent robotic capability.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics, vlms, locomotion, manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nZsqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a systematic evaluation of vision-language models (VLMs) for robotic control. The key contribution is a comparative framework that investigates how different control interfaces affect VLM performance. The authors examine four interfaces: Direct Control, Code Control, Policy Supervision, and Training Supervision. Through extensive experiments spanning classical control, locomotion, and manipulation tasks in simulation, the study concludes that the choice of control interface plays a critical role in performace of VLMs for robotic control.

Critical strengths
  • The topic is valuable that evaluate VLM's capabilities while considering control interfaces and external resources.
  • The evaluated models and experiments are sufficiently comprehensive.
Critical weaknesses
  • The results are somewhat expected, as the four interfaces differ in their levels of abstraction and associated costs, and the models vary in their capabilities.
  • Although the authors acknowledge that the paper lacks real-world experiments in the limitations, and the results may differ from those in simulation due to noisy or inaccurate observations in real-world settings.
Questions for authors
  • Why are not all tasks evaluated using all four interfaces? In particular, why are Code Control and Training Supervision not evaluated on manipulation tasks?
  • MolmoAct achieves an 86% success rate on LIBERO-40 when run alone, but performance decreases with VLM supervision. How often does the VLM accept, reject, or modify MolmoAct’s actions? Could the authors analyze how these interventions affect performance?
  • The results may depend more on model capabilities than on control interfaces. In my tests, newer models like GPT-6 Astra can directly control robots. Could the authors analyze whether the advantages of Code Control and Policy Supervision diminish as model capabilities improve?
  • It is confusing that the G1 fails to stand up by training in Table 4, as many open-source repositories have demonstrated that this skill can be successfully trained. Could the authors provide more details about the experimental setup?
4 comments

Thank you for reviewing our paper and providing thoughtful critique. We respond below. In short, we explain why our broad framing allows us to make generalizations that more specific studies do not, and analyze the trajectory of model performance over time.

(W1) The results are not surprising because the interface abstractions need different skills and models possess different skills \textbf{(W1) The results are not surprising because the interface abstractions need different skills and models possess different skills } Our results show differences that cannot be predicted from overall model capability or interface complexity alone. Our work compares four common ways of controlling robots to identify which abilities models possess and where they struggle (Section 1, lines 39–50).

For instance, the relative ranking of models changes across interfaces. On the quadruped standing task, Opus 4.7 gets 94% success under Code Control but 22% under Training Supervision, while GPT-5.4 achieves 48.6% and 74.3%, respectively (Table 4). Performance also differs substantially across tasks: newer models can make a quadruped walk using code but still struggle with direct manipulation on LIBERO, where the best model achieves only 5.5% success (Figure 3; Appendix Figure 17).

Prior research has often focused on precursor skills such as vision and scene understanding, or studied individual control interfaces separately (Section 2, lines 99–117). We expand this research by testing how models use these abilities to complete robotic tasks. For instance, our findings about simple aids helping more than dense visual aids show that more information is not necessarily better: a compass improves Go2 performance more than all tested aids combined, while targeted 3D queries help manipulation more than depth or segmentation overlays (Figures 5 and 7). Together, these results help identify which capabilities and tools matter most and where further research is needed in the field, and would be difficult to ascertain from a more specific study.

(W2) The experiments are in simulation, which limits generalization to the real world \textbf{(W2) The experiments are in simulation, which limits generalization to the real world } We aim to test if VLMs can control robots without being limited by their response speed, which is not currently possible outside of simulation environments.

For example, our Go2 and G1 experiments apply commands every 12 milliseconds, much faster than current VLMs can respond (Appendix A.1, lines 809–810). We therefore pause the simulator while the model responds. This lets us test their control ability separately from their speed (Section 3, lines 145–149). Simulation also lets us compare models under the same physical conditions within each task.While it is possible that simulation results do not transfer to real life, other work has successfully transferred simulation-trained policies to physical robots (for example: DrEureka, Ma et al., 2024). We acknowledge the limits of simulation in the paper and will expand in the limitations section (Appendix J, lines 1667–1670).

(Q1) Why are the tasks not all evaluated on the four interfaces? Particularly, why are Code Control and Training Supervision not evaluated on manipulation tasks?\textbf{(Q1) Why are the tasks not all evaluated on the four interfaces? Particularly, why are Code Control and Training Supervision not evaluated on manipulation tasks?} We prioritized combinations that were informative, commonly used, and feasible within our computational budget (Section 3.1, lines 208–213).

We did not evaluate Code Control on manipulation because Direct Control already uses an underlying arm controller to execute movements (Appendix A.1, lines 821–824). We also conducted 10 preliminary LIBERO trials using Python controllers and did not find clear improvements over Direct Control, as both failed to solve the tasks. Given these results, we prioritized a full evaluation of Direct Control rather than extending Code Control, especially because this is a common interface in robotics papers.

We did not evaluate Training Supervision for manipulation because the 40 LIBERO tasks would likely require separate training runs in our setup which would be very costly. Unlike our other training experiments, which use simulator positions and velocities, manipulation would require training policies that process images, substantially increasing computational cost (Appendix A.2, lines 847–863). We considered this beyond our available budget.

For other omissions, pretrained policies are uncommon for simple classic-control tasks, while we used Go2 to evaluate high-level policy-guided locomotion rather than repeating similar experiments on G1. We will add these selection reasons and the resulting limitations to Appendix A.

(Q2) How often do VLMs accept, reject, or modify MolmoAct’s actions, and how do these interventions affect performance?\textbf{(Q2) How often do VLMs accept, reject, or modify MolmoAct’s actions, and how do these interventions affect performance?} We give the acceptance rates in Figure 6b, but will re-label it to make this clearer. VLM supervision can reduce success because VLMs sometimes edit or decline the pretrained policy's commands despite being less successful than it at direct control. MolmoAct alone achieves 86% success on LIBERO-40, outperforming all supervised VLMs (Figure 6a). The best-performing supervisors generally rely more on MolmoAct's proposed actions (Figure 6b), suggesting that their unnecessary interventions hurt performance.

However, we also show that newer models are more selective when the pretrained policy is less proficient. On novel tasks outside MolmoAct's training distribution, Opus 4.6 reduces its use of MolmoAct actions from 77% to 27% of simulator steps, compared with 31% to 26% for Opus 4 (Appendix Figure 18). Some models even complete tasks where MolmoAct alone fails (Section 4.3, lines 409–414). Thus, intervening can be useful, but more intervention does not necessarily improve success.

(Q3) Could the authors analyze whether the advantages of Code Control and Policy Supervision diminish as model capabilities improve? \textbf{(Q3) Could the authors analyze whether the advantages of Code Control and Policy Supervision diminish as model capabilities improve? } Our results do not show that these advantages diminish as models improve. For example, Code Control's advantage over Direct Control on Go2 stand-up grows from 5.7 percentage points for Opus 4.1 to 96 points for Mythos Preview (Table 4). For manipulation, Policy Supervision's advantage shrinks from 72.5 points for Opus 4.6 to 29 points for Mythos Preview. This is mainly because supervised performance falls, not because Direct Control improves (Figure 6, Appendix Figure 17).

We suspect that as model capabilities improve, both interfaces will remain useful. Code Control allows robots to execute many actions without waiting for another model response. Recent work supports its usefulness even for newer models. PyRUA-Lean finds that GPT-6 Astra achieves 71.7% success when using Python to combine robot tools, compared with 63.1% using ordinary tool calls (Si et al., 2026, “Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens”).

Policy Supervision is also useful for newer models on some tasks. Other researchers found that GPT-6 Astra achieves 48% success with policy assistance versus 26% with Direct Control; however, Direct Control performs better on RoboLab (Galbot Team, 2026, “Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies”). Stronger models may need less assistance on some tasks, but can still benefit from pretrained policies on others.

We will add this discussion to Section 5 and identify comparisons with newer models as future work in the conclusion.

(Q4) Previous repositories have trained the G1. Why couldn’t it be done here, and can the authors give more details on the experimental setup? (Q4)\textbf{(Q4) Previous repositories have trained the G1. Why couldn’t it be done here, and can the authors give more details on the experimental setup? (Q4)} It couldn’t be done here because of lack of skill and experimental dexterity on the VLMs we tested, even though other repositories can train the model with less time on fewer resources. Mythos, Opus 4.7, GPT 5.4 were able to train easier embodiments in classic control and on the quadruped robots with the same interface (Figures 2 & 4). We provided a PPO trainer using MuJoCo Warp on H200 GPUs.The VLM must define the rewards and starting states, choose the networks and hyperparameters, and decide how many steps to train, and what the different phases of training are— a substantially complex undertaking. It can revise these choices using training statistics and deployment feedback and retrain policies. During evaluation, G1 starts on its side and must remain upright for two seconds, but no model managed to get it upright for more than a quarter of a second (Appendix A.1, lines 810–811). Our anonymized code with more specific details is available in the paper (line 89).

Successful training has been accomplished before, but it usually requires a litany of specialized rewards and training procedures. For instance, the public g1-humanoid-rl repository claims 30–50 minutes per PPO training attempt on a DGX Spark— which is substantially less powerful than an H200— with successful stand-up after four rounds of reward and termination changes (Chapter 14). However, the VLMs must make these training decisions themselves, and this was beyond the capabilities of the models we tested.

Thank you for the response. It has addressed some of my concerns. Considering the paper's contributions, I will maintain my original score.

Thank you for reading and your candid response. What questions or concerns do you still have that could change your evaluation of the paper?

Regarding your concern that the results are unsurprising or only reflect overall model capability, we highlight four findings that we view as valuable and unexpected:

  • Newer VLMs are not always better at robotics. Opus 4.7 beats Mythos Preview on manipulation Policy Supervision (51% vs. 34.5%). We also identify a causal mechanism for this: Mythos also uses fewer MolmoAct actions (31% vs. 61% of steps), suggesting that it is overconfident in its ability to improve actions (Section 4.3, lines 402–407; Appendix Figure 18).
  • Overall model strength cannot predict task performance. GPT-5.4 beats Opus 4.7 on Go2 stand-up under Training Supervision (74.3% vs. 22%), but loses under Code Control (48.6% vs. 94%). Model strength cannot predict which model is best, although it is correlated with strength across all tasks (Table 4, lines 1007–1018).
  • Interfaces improve at different rates. Abilities are improving for each generation, but the relative speed is not known. We find Code Control and Policy Supervision have improved faster than Direct Control. This will be especially interesting to analyze in light of newer generations, which are likely trained on robotics data, to see if this trend continues (Section 5, lines 454–460).
  • VLMs recognize capable policies even when their success is very low. Opus 4.6 reduces its use of MolmoAct from 77% of steps on familiar tasks to 27% on novel tasks where MolmoAct fails, despite never succeeding, and other VLMs perform similarly (Section 4.3, lines 409–414; Appendix Figure 18).
Reviewer_iz4gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores how well VLMs can serve as components in simulated robot control systems. It evaluates general-purpose VLMs across four control interfaces: direct action output, code generation, calling pretrained robot policies, and supervising robot training. The results show that model performance depends heavily on the tools and interfaces provided. Having VLMs write control code or call existing robot policies generally works better than directly controlling robot actions. Current models can already handle some quadruped locomotion and manipulation tasks, but still struggle to reliably control complex humanoid robots. Overall, VLMs are better at understanding the environment, planning tasks, and making high-level decisions, while fine-grained dynamics control still requires specialized robot control algorithms.

Critical strengths
  1. The research question is valuable. It asks whether general-purpose VLMs have the ability to control the physical world. The paper systematically evaluates model capabilities through four control interfaces. The categorization is reasonable and covers different application modes, from direct action output to calling specialist policies and automatic policy training. This avoids a one-sided evaluation that relies on a single experimental setting. The results not only demonstrate VLM's potential robotic capabilities but also clearly reveal its current capability boundary.

  2. The experiments are fairly comprehensive. They don't just look at whether models can complete tasks, but also analyze why certain methods work better. The paper compares different approaches such as direct control, code writing, and calling existing policies. It finds that when VLMs are given the right tools and information, their robotic capabilities improve significantly.

Critical weaknesses
  1. GPT-6 already shows significant advantages in robot control and complex task planning. The paper may have been unable to include the latest models due to time constraints. I recommend adding experiments with frontier models like GPT-6 to better assess the current capability boundary of VLMs.

  2. There is a lack of real-world hardware evaluation, so the conclusions are limited by simulation and cannot demonstrate real-world robotic capability. The evaluation metrics focus heavily on task success rate, with insufficient analysis of control quality, such as task timing. The current evaluation mostly covers short-horizon tasks, with limited assessment of long-horizon tasks and memory tasks.

Questions for authors

My question is the same as the issue raised in the weaknesses.

3 comments

Thank you for your thoughtful review and kind words. We respond to your weaknesses here, and hope they help better explain our paper and its scope. Please let us know if you have any other questions!

(W1) Can you test GPT-6 and other newer models to test the current boundary of ability?\textbf{(W1) Can you test GPT-6 and other newer models to test the current boundary of ability?} We view extending the evaluation as future work given the superior capabilities of newer models and the fact that there is some recent indication that newer models were specifically trained on robotics data. (See the public Github repository “vlm-robotics-training”) Our evaluation is focused on a specific snapshot of models (Appendix C.1, Table 3). These tasks are tailor-mode to understand how general code and vision competency translates into robotics domains, where understanding kinematics and scenes is very important, and not robotics-specific training (Section 1, lines 83–91).

Our experiments are also extremely expensive to run: some of our trials approach 1,000,000 tokens, and each Training Supervision session allows sessions of up to four hours on H200 GPUs (Section 3, lines 135–136; Appendix A.2, lines 853–855). It would not be practical to run additional models on more than a few experiments. We acknowledge that these tests could provide useful task-specific comparisons, but would not update the full analysis across tasks and interfaces.

We will identify evaluation of newer models, and the development of tasks to distinguish how they capability differs, as future work in the Conclusion.

(W2.1) There is no evaluation on hardware and thus you cannot demonstrate real-world robotic capability \textbf{(W2.1) There is no evaluation on hardware and thus you cannot demonstrate real-world robotic capability } Our goal is to test whether current VLMs have the ability to control robots without being limited by their response speed, which is not possible on real hardware.

For example, our Go2 and G1 experiments apply commands every 12 milliseconds, much faster than current VLMs can respond (Appendix A.1, lines 809–810). We therefore pause the simulator while the model responds. This lets us test their control ability even when their speed would prevent real-time operation (Section 3, lines 145–149). As VLMs get faster, these results will become more relevant to real-world control. For other experiments, running them all in simulation also allows more consistent comparisons between interfaces.

While it is possible that simulation results do not transfer to real life, other work has successfully transferred simulation-trained policies to physical robots (for one example: DrEureka, Ma et al., 2024). Thus, we believe our results have real-world relevance, although they do not establish that our controllers would succeed on physical hardware. We acknowledge differences in real-world sensors and simulated contact dynamics/kinematics in Appendix J (lines 1667–1670).

(W2.2)The paper analyzes accuracy but not other relevant metrics such as task timing \textbf{(W2.2)The paper analyzes accuracy but not other relevant metrics such as task timing } We agree that task timing is important, but wall-clock timings in our experiments would be difficult to interpret because they depend heavily on API latency and our relatively unoptimized implementations.

We focused on measuring robotic control ability without response speed limiting performance. Current VLMs are too slow to control high-frequency robots directly in real time, so we pause the simulator while they respond (Section 3, lines 145–149; Appendix A.1, lines 779–783). We also already measure aspects of control quality, such as balance duration and ball airtime (Table 1, lines 166–176). However, we agree that measuring how quickly models complete tasks in simulated time would provide additional insight.

(W2.3)The paper mostly covers short horizon tasks, not long horizon tasks \textbf{(W2.3)The paper mostly covers short horizon tasks, not long horizon tasks } We have some longer tasks in LIBERO, but mainly cover short tasks because longer ones would overflow the context limits and because these models still struggle with basic control even over short horizons. For example, every tested model has 0% success on G1 stand-up (Table 4), and the best Direct Control success on LIBERO-40 is 5.5%, achieved by Mythos Preview (Figure 17). Longer tasks that require these skills could produce too few successes to distinguish models or be of any practical use. We also observe that for direct control especially, long experiments overflow context limits (lines 267–269).

We already include a few long-horizon tasks, because LIBERO-40 includes tasks from the long-horizon part of the benchmark, although they are not so much longer than the other tasks (Appendix A.1, lines 826–827). Our Go2 tasks also test remembering color sequences and recalling layouts, which are much longer than the other tasks we test (Appendix E, lines 1145–1148).

Thank you for the detailed responses and clarifications. The explanations regarding the computational cost, simulation setup, and limitations of real-world evaluation are helpful. I also appreciate the clarification that the experiments include some longer-horizon tasks and control-quality metrics beyond task success rates. I agree that evaluating newer frontier models and conducting real-world experiments would be valuable future directions.

Thank you! Your thoughtful questions were valuable. We would really appreciate if you could consider raising your score in light of our response.

Reviewer_MnyzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study examines if general purpose vision-language models can apply their current capabilities to control robots. Four different methods are studied; generating actions directly, creating control programs (code), controlling a pre-trained model (policy), and developing both code and reward structures to train a new model (policy) via reinforcement learning. These tasks include basic control, robot standing and walking, and object manipulation. All studies were conducted in simulation. In addition to studying various types of models and versions, this study also evaluated if additional inputs such as compass direction and depth map would improve performance. Results indicate that the type of interface and tools available to the user will have a large impact on success. In many cases, writing code or using a pre-trained policy was more successful than direct control. Additionally, providing more input did not always result in better performance.

Critical strengths
  1. This study evaluates four control methods for various robot tasks and models to provide an overview of the performance of general-purpose VLMs in different environments.
  2. The results demonstrate that the selection of tools and inputs will greatly influence the performance. For example, using compass directions and target 3D queries may be more beneficial than incorporating multiple types of visual inputs.
Critical weaknesses
  1. The scope is too large, each of the four approaches tests a different set of skills (e.g., action selection, writing RL training code). Each approach requires more in-depth research to determine what the model can and cannot do.
  2. It is difficult to compare the results. The settings vary in terms of input data, available tools, amount of computation used, and number of times a trial can be retried. For example, some Direct trials end when a failure occurs but Code trials can be retried. It's difficult to isolate the effects of the control interface from the effects of additional resources.
  3. While using the same prompt for all models may seem reasonable, there is no guarantee that the prompt will work as effectively for each model.
  4. Policy Supervision is dependent upon the pre-trained policy and how the VLM interacts with the pre-trained policy. Training Supervision is dependent upon the provided trainer and training tools. So the final score is also dependent on those predefined parts by the authors for the specific interface, and it's hard to conclude on the general kind of interface.
  5. Explanations for unexpected results are needed, for example, some newer models sometimes perform worse than older models and VLM supervision can decrease the success rate of a pre-trained policy, where further research is required to explain why they occur and give more insights.
Questions for authors

See weakness.

3 comments

Thank you for your thoughtful review and close reading. We respond to your comments below and in other comments, particularly justifying our scope and the specific insight we glean from it and the comparisons we hope to inspire.

(W1) The paper’s scope is too large because the four approaches test different skills, each requiring more in-depth research to determine what models can and cannot do \textbf{(W1) The paper's scope is too large because the four approaches test different skills, each requiring more in-depth research to determine what models can and cannot do }

Our paper provides analysis that is only possible when multiple embodiments and interfaces are involved. As the space of possible harnesses and prompts is effectively unbounded, our goal is not to find the absolute ceiling on performance. Instead, we identify several common paradigms, and our experiments give baselines for what models can achieve under these settings (Section 1, lines 39-50, 83-91).

We are concerned with patterns that emerge from comparisons across interfaces, tasks, and model generations. For instance, Section 5 shows that there is no fixed performance ordering among interfaces (lines 450-453), and that Code Control and Policy Supervision generally improve faster across model generations than Direct Control (lines 454-460). Across locomotion and manipulation, targeted information helps more than dense visual overlays (Figures 5 and 7; lines 467-472). We also find that VLMs rely less on pretrained policies on unfamiliar tasks, and occasionally complete tasks where the underlying policy fails, although such successes remain rare (Section 4.3, lines 409-414). These cross-task findings help identify which capabilities are developing and where further investigation is warranted.

Our results are broad and not intended as a complete characterization of capability under each interface. We will add these caveats to the limitations section.

(W2) Differences in tools, computation, and retries make the interfaces difficult to compare\textbf{(W2) Differences in tools, computation, and retries make the interfaces difficult to compare} We prioritized realistic deployments over artificially identical settings; our goal is to compare several common paradigms and give baselines and analysis for what models achieve under these settings (Section 1, lines 3950). For example, we use end-effector commands for manipulation instead of matching raw torque to our humanoid/quadruped experiments because this better reflects how robotic arms are commonly controlled (Appendix A.1, lines 809–824). We also give different amounts of cluster time to train Classic Control and more complex embodiments because they require different amounts of time to train effectively.

However, our comparisons are still meaningful because we evaluate multiple interfaces on the same tasks using the same success criteria (Tables 1–2). In classic control, Direct, Code, and deployed Training controllers also use the same action frequency (Appendix A.1, lines 785–790). We matched as much as was possible per-task. (Section 3, lines 135–142).

The main difference the reviewer identifies—that Code Control can retry failed stand-up attempts while Direct Control cannot—was a modification we made to allow both approaches opportunities for correction. If we did not allow retries, the VLM would not get to self-correct on Code Control; on Direct, because it only gives a single command at a time, it can see how the robot is moving and make dynamic adjustments. We also provide the first-attempt performance in order to compare first-attempt performance between code and direct (Appendix Figure 22) .

(W3) Using the same prompt for all models is unfair because it may work better for some models than others\textbf{(W3) Using the same prompt for all models is unfair because it may work better for some models than others} We agree that a shared prompt is not optimal for each model, but we chose this for a standardized comparison rather than optimizing each model individually. Prior work does this as well, such as VLMEvalKit, which uses shared prompts by default in VLMEvalKit (Duan et al., 2024, “VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models”).

Additionally, before starting our trials, we conducted small preliminary tests on Anthropic, OpenAI, and Qwen models with multiple prompts and chose the one that maximized overall success rather than performance for any individual model. This informed the phrasing we chose such as “You will look over its actions and use them to perform the task” (Appendix B, lines 915–917). Empirically, we did not observe clear performance differences from the order changes or small wording changes we tested. Prior work also finds that more recent, instruction-tuned language models can be less sensitive to rephrasing, although there is still variation (Zhou et al., 2024, “Larger and more instructable language models become less reliable”).

Exhaustively optimizing prompts for every model was not feasible under our compute constraints, so our results should be interpreted as average performance under a best-effort prompt. (Appendix J, lines 1660–1663).

(W4) The Policy and Training Supervision results depend on the specific pretrained policies and harnesses, making it difficult to generalize to these interfaces. \textbf{(W4) The Policy and Training Supervision results depend on the specific pretrained policies and harnesses, making it difficult to generalize to these interfaces. } We agree that our results cannot speak for all harnesses and models, but our implementations use common interaction patterns and training procedures that aren’t unique to the components we chose.

In Policy Supervision, we use a proficient pretrained policy (MolmoAct achieves 86% success on LIBERO-40, Figure 6), but the VLM interacts with its proposed end-effector actions, not its internal architecture (Section 3, lines 155–158; Appendix B, lines 913–932). Other VLAs, such as OpenVLA, produce similar action commands, so this supervision mechanism could be applied identically to them as well (Kim et al., 2024, “OpenVLA: An Open-Source Vision-Language-Action Model”). We also evaluate a separate Go2 walking policy through high-level velocity commands (Appendix E, lines 1136–1140).

We agree that there are many possible training setups and that models will have different degrees of proficiency with them. Our setup uses standard PPO, as in Eureka, with a fixed time budget, but leaves substantial control to the VLM (Ma et al., 2024, “Eureka: Human-Level Reward Design via Coding Large Language Models”). It specifies the policy and value networks, rewards, curricula, hyperparameters, and how many environment steps to train for in each call (Appendix A.2, lines 847–863). Our setup induces a gradient of performance (Figures 2 and 4), demonstrating differences in models' ability to use this common training procedure. We view it as future work to determine which aspects of training can make this task harder or easier.

(W5.1) Can you explain why some newer models are worse than older models? \textbf{(W5.1) Can you explain why some newer models are worse than older models? } Newer models sometimes perform worse than older models due to differences in what they are trained to do and their relative strengths, although we cannot isolate all causes in our experiments. For instance, prior work has found that newer models do not necessarily improve performance on every skill (Chen et al., 2024 “How Is ChatGPT’s Behavior Changing Over Time”).

We also observe behavioral differences that help explain some regressions. For example, Mythos Preview performed worse than Opus 4.7 on Policy Supervision for manipulation (34.5% versus 51% success; Section 4.3, lines 402–405). However, Mythos also relied less on the pretrained MolmoAct policy: only 31% of its executed simulator steps used MolmoAct's actions, compared with 61% for Opus 4.7 (Appendix Figure 18). This may also help explain why VLM supervision can decrease success: MolmoAct alone succeeds on 86% of LIBERO-40 tasks (Figure 6), so replacing its actions with less effective ones could hurt performance. These results suggest a possible explanation, although they do not establish whether individual interventions caused the failures or why the models differed in their reliance on MolmoAct.

We will add this discussion to Section 4.3 and identify analyzing when VLMs accept, modify, or replace specialist actions, and whether those interventions improve outcomes, as future work.

Here we finish responding to your comments. Please let us know if anything is unclear or if you have any further concerns, and we hope you will consider our clarifications.

(W5.2) Can you explain why VLM supervision can reduce the success rate for a pretrained policy? \textbf{(W5.2) Can you explain why VLM supervision can reduce the success rate for a pretrained policy? } VLM supervision can reduce success because they sometimes edit or decline the pretrained policy's commands despite being less proficient than it at direct control, as shown in Figure 6b. MolmoAct alone achieves 86% success on LIBERO-40, outperforming all VLMs we tested (Figure 6a). The best-performing supervisors generally rely more on MolmoAct's proposed actions (Figure 6b), which suggests that unnecessary edits/drops hurt performance.

However, we also show that newer models are more selective when the pretrained policy is less proficient. On novel tasks outside MolmoAct's training distribution, Opus 4.6 reduces its use of MolmoAct actions from 77% to 27% of simulator steps, compared with 31% to 26% for Opus 4 (Appendix Figure 18). Some models even complete tasks where MolmoAct alone fails (Section 4.3, lines 409–414). Thus, intervening can be useful, but current VLMs do not reliably distinguish helpful from harmful interventions.

We will make this discussion more prominent in Section 4.3 and fix Figure 6b's label to be clearer.

(W5.3) Can you discuss where future research is required? \textbf{(W5.3) Can you discuss where future research is required? } Future work is needed to determine the causal mechanisms behind the effects we find—particularly why newer models sometimes perform worse and why VLM supervision can reduce a pretrained policy's success—and whether these effects can be fixed. For example, comparing rollouts with and without VLM interventions from the same simulator states could help determine when interventions improve or harm performance, rather than merely observing differences in policy reliance (Section 4.3, lines 402–414). See (W5.1) for discussion for

More broadly, we want to understand how these findings change as models begin receiving more robotics-specific training. Will they generalize across hardware, or become specialized to the robots on which they were trained? As we get faster VLMs, will their performance in simulation generalize to physical robots, where sensing and actuation differ (Appendix J, lines 1667–1670)? Finally, why does practice improve performance on repeated routes, while retaining more manipulation history provides limited benefits, and can methods that better leverage experience (such as compaction) remedy this? (Appendix H, lines 1450–1454, 1478–1484).

We will add these specific future directions and the unresolved questions they address to the conclusion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Dear AC and Reviewers, we sincerely appreciate your review of our paper and our shared discussion of it. We have posted a revision copy incorporating the review comments. For your convenience, we summarize the discussion below.

Consensus on Strengths

Despite initially mixed scores (2/3/2), the reviewers identified strengths in the research question, the breadth of our experiments, and our findings on the effect of tools and inputs.

  • Research Value: Reviewer iz4g wrote that “The research question is valuable. It asks whether general-purpose VLMs have the ability to control the physical world.” and Reviewer nZsq similarly said that “The topic is valuable that evaluate VLM's capabilities while considering” other factors. Reviewer Mnyz mentioned our setup and different environments under Critical Strengths
  • Comprehensive Comparative Evaluation: Reviewer Mnyz wrote: “This study evaluates four control methods for various robot tasks and models to provide an overview of the performance of general-purpose VLMs in different environments.” Reviewer iz4g added: “The experiments are fairly comprehensive. They don't just look at whether models can complete tasks, but also analyze why certain methods work better.” Reviewer nZsq also described the evaluated models and experiments as sufficiently comprehensive.
  • Practical Insights for Robot Control: Reviewer Mnyz wrote that “The results demonstrate that the selection of tools and inputs will greatly influence the performance. For example, using compass directions and target 3D queries may be more beneficial than incorporating multiple types of visual inputs.” Reviewer iz4g also highlighted our finding that giving VLMs the right tools and information substantially improves their robotic capabilities.

Reviewer Summary

Reviewer iz4g raised concerns about the models we chose to evaluate and questioned why we limited our evaluation to simulation. We clarified that newer models are qualitatively different (both because of their training material and capabilities) and that many of our experiments are impossible on real-hardware and conducting all in simulation is maximally comparable. They said that “The explanations regarding the computational cost, simulation setup, and limitations of real-world evaluation are helpful.” and agreed with our points about newer frontier models.

We addressed some of Reviewer nZsq's concerns about the experimental setup and explanations behind behaviour, and they said that our clarification “has addressed some of my concerns.” but had lingering doubts about our contribution. We responded with the four most surprising results from our work: that Newer VLMs are not always better at robotics, that raw model strength does not at all predict performance on all of our tasks, that the interfaces we test are not improving at the same rate through model generations, and that VLMs recognize capable policies even when their success is very low. Although we have not received a response, we hope they will recognize these contributions as significant and novel.

While we did not receive a followup from Reviewer Mnyz, we believe we have justified our large scope by only making claims that result from our large-scale comparison; explaining any differences in the different interfaces and the reason we formulated the experiments that way; justified our prompt usage with prior work and our preliminary studies; gave explanation for why the method of interaction with the policy is general and adding a caveat to the Limitations section of the paper; and identified in our paper’s analysis of why newer models can perform worse than old ones.

We hope our summary is helpful for the remainder of the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.