- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a systematic evaluation of vision-language models (VLMs) for robotic control. The key contribution is a comparative framework that investigates how different control interfaces affect VLM performance. The authors examine four interfaces: Direct Control, Code Control, Policy Supervision, and Training Supervision. Through extensive experiments spanning classical control, locomotion, and manipulation tasks in simulation, the study concludes that the choice of control interface plays a critical role in performace of VLMs for robotic control.
Critical strengths
- The topic is valuable that evaluate VLM's capabilities while considering control interfaces and external resources.
- The evaluated models and experiments are sufficiently comprehensive.
Critical weaknesses
- The results are somewhat expected, as the four interfaces differ in their levels of abstraction and associated costs, and the models vary in their capabilities.
- Although the authors acknowledge that the paper lacks real-world experiments in the limitations, and the results may differ from those in simulation due to noisy or inaccurate observations in real-world settings.
Questions for authors
- Why are not all tasks evaluated using all four interfaces? In particular, why are Code Control and Training Supervision not evaluated on manipulation tasks?
- MolmoAct achieves an 86% success rate on LIBERO-40 when run alone, but performance decreases with VLM supervision. How often does the VLM accept, reject, or modify MolmoAct’s actions? Could the authors analyze how these interventions affect performance?
- The results may depend more on model capabilities than on control interfaces. In my tests, newer models like GPT-6 Astra can directly control robots. Could the authors analyze whether the advantages of Code Control and Policy Supervision diminish as model capabilities improve?
- It is confusing that the G1 fails to stand up by training in Table 4, as many open-source repositories have demonstrated that this skill can be successfully trained. Could the authors provide more details about the experimental setup?
4 comments
We give the acceptance rates in Figure 6b, but will re-label it to make this clearer. VLM supervision can reduce success because VLMs sometimes edit or decline the pretrained policy's commands despite being less successful than it at direct control. MolmoAct alone achieves 86% success on LIBERO-40, outperforming all supervised VLMs (Figure 6a). The best-performing supervisors generally rely more on MolmoAct's proposed actions (Figure 6b), suggesting that their unnecessary interventions hurt performance.
However, we also show that newer models are more selective when the pretrained policy is less proficient. On novel tasks outside MolmoAct's training distribution, Opus 4.6 reduces its use of MolmoAct actions from 77% to 27% of simulator steps, compared with 31% to 26% for Opus 4 (Appendix Figure 18). Some models even complete tasks where MolmoAct alone fails (Section 4.3, lines 409–414). Thus, intervening can be useful, but more intervention does not necessarily improve success.
Our results do not show that these advantages diminish as models improve. For example, Code Control's advantage over Direct Control on Go2 stand-up grows from 5.7 percentage points for Opus 4.1 to 96 points for Mythos Preview (Table 4). For manipulation, Policy Supervision's advantage shrinks from 72.5 points for Opus 4.6 to 29 points for Mythos Preview. This is mainly because supervised performance falls, not because Direct Control improves (Figure 6, Appendix Figure 17).
We suspect that as model capabilities improve, both interfaces will remain useful. Code Control allows robots to execute many actions without waiting for another model response. Recent work supports its usefulness even for newer models. PyRUA-Lean finds that GPT-6 Astra achieves 71.7% success when using Python to combine robot tools, compared with 63.1% using ordinary tool calls (Si et al., 2026, “Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens”).
Policy Supervision is also useful for newer models on some tasks. Other researchers found that GPT-6 Astra achieves 48% success with policy assistance versus 26% with Direct Control; however, Direct Control performs better on RoboLab (Galbot Team, 2026, “Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies”). Stronger models may need less assistance on some tasks, but can still benefit from pretrained policies on others.
We will add this discussion to Section 5 and identify comparisons with newer models as future work in the conclusion.
It couldn’t be done here because of lack of skill and experimental dexterity on the VLMs we tested, even though other repositories can train the model with less time on fewer resources. Mythos, Opus 4.7, GPT 5.4 were able to train easier embodiments in classic control and on the quadruped robots with the same interface (Figures 2 & 4). We provided a PPO trainer using MuJoCo Warp on H200 GPUs.The VLM must define the rewards and starting states, choose the networks and hyperparameters, and decide how many steps to train, and what the different phases of training are— a substantially complex undertaking. It can revise these choices using training statistics and deployment feedback and retrain policies. During evaluation, G1 starts on its side and must remain upright for two seconds, but no model managed to get it upright for more than a quarter of a second (Appendix A.1, lines 810–811). Our anonymized code with more specific details is available in the paper (line 89).
Successful training has been accomplished before, but it usually requires a litany of specialized rewards and training procedures. For instance, the public g1-humanoid-rl repository claims 30–50 minutes per PPO training attempt on a DGX Spark— which is substantially less powerful than an H200— with successful stand-up after four rounds of reward and termination changes (Chapter 14). However, the VLMs must make these training decisions themselves, and this was beyond the capabilities of the models we tested.
Thank you for the response. It has addressed some of my concerns. Considering the paper's contributions, I will maintain my original score.
Thank you for reading and your candid response. What questions or concerns do you still have that could change your evaluation of the paper?
Regarding your concern that the results are unsurprising or only reflect overall model capability, we highlight four findings that we view as valuable and unexpected:
- Newer VLMs are not always better at robotics. Opus 4.7 beats Mythos Preview on manipulation Policy Supervision (51% vs. 34.5%). We also identify a causal mechanism for this: Mythos also uses fewer MolmoAct actions (31% vs. 61% of steps), suggesting that it is overconfident in its ability to improve actions (Section 4.3, lines 402–407; Appendix Figure 18).
- Overall model strength cannot predict task performance. GPT-5.4 beats Opus 4.7 on Go2 stand-up under Training Supervision (74.3% vs. 22%), but loses under Code Control (48.6% vs. 94%). Model strength cannot predict which model is best, although it is correlated with strength across all tasks (Table 4, lines 1007–1018).
- Interfaces improve at different rates. Abilities are improving for each generation, but the relative speed is not known. We find Code Control and Policy Supervision have improved faster than Direct Control. This will be especially interesting to analyze in light of newer generations, which are likely trained on robotics data, to see if this trend continues (Section 5, lines 454–460).
- VLMs recognize capable policies even when their success is very low. Opus 4.6 reduces its use of MolmoAct from 77% of steps on familiar tasks to 27% on novel tasks where MolmoAct fails, despite never succeeding, and other VLMs perform similarly (Section 4.3, lines 409–414; Appendix Figure 18).
Thank you for reviewing our paper and providing thoughtful critique. We respond below. In short, we explain why our broad framing allows us to make generalizations that more specific studies do not, and analyze the trajectory of model performance over time.
(W1) The results are not surprising because the interface abstractions need different skills and models possess different skills Our results show differences that cannot be predicted from overall model capability or interface complexity alone. Our work compares four common ways of controlling robots to identify which abilities models possess and where they struggle (Section 1, lines 39–50).
For instance, the relative ranking of models changes across interfaces. On the quadruped standing task, Opus 4.7 gets 94% success under Code Control but 22% under Training Supervision, while GPT-5.4 achieves 48.6% and 74.3%, respectively (Table 4). Performance also differs substantially across tasks: newer models can make a quadruped walk using code but still struggle with direct manipulation on LIBERO, where the best model achieves only 5.5% success (Figure 3; Appendix Figure 17).
Prior research has often focused on precursor skills such as vision and scene understanding, or studied individual control interfaces separately (Section 2, lines 99–117). We expand this research by testing how models use these abilities to complete robotic tasks. For instance, our findings about simple aids helping more than dense visual aids show that more information is not necessarily better: a compass improves Go2 performance more than all tested aids combined, while targeted 3D queries help manipulation more than depth or segmentation overlays (Figures 5 and 7). Together, these results help identify which capabilities and tools matter most and where further research is needed in the field, and would be difficult to ascertain from a more specific study.
(W2) The experiments are in simulation, which limits generalization to the real world We aim to test if VLMs can control robots without being limited by their response speed, which is not currently possible outside of simulation environments.
For example, our Go2 and G1 experiments apply commands every 12 milliseconds, much faster than current VLMs can respond (Appendix A.1, lines 809–810). We therefore pause the simulator while the model responds. This lets us test their control ability separately from their speed (Section 3, lines 145–149). Simulation also lets us compare models under the same physical conditions within each task.While it is possible that simulation results do not transfer to real life, other work has successfully transferred simulation-trained policies to physical robots (for example: DrEureka, Ma et al., 2024). We acknowledge the limits of simulation in the paper and will expand in the limitations section (Appendix J, lines 1667–1670).
(Q1) Why are the tasks not all evaluated on the four interfaces? Particularly, why are Code Control and Training Supervision not evaluated on manipulation tasks? We prioritized combinations that were informative, commonly used, and feasible within our computational budget (Section 3.1, lines 208–213).
We did not evaluate Code Control on manipulation because Direct Control already uses an underlying arm controller to execute movements (Appendix A.1, lines 821–824). We also conducted 10 preliminary LIBERO trials using Python controllers and did not find clear improvements over Direct Control, as both failed to solve the tasks. Given these results, we prioritized a full evaluation of Direct Control rather than extending Code Control, especially because this is a common interface in robotics papers.
We did not evaluate Training Supervision for manipulation because the 40 LIBERO tasks would likely require separate training runs in our setup which would be very costly. Unlike our other training experiments, which use simulator positions and velocities, manipulation would require training policies that process images, substantially increasing computational cost (Appendix A.2, lines 847–863). We considered this beyond our available budget.
For other omissions, pretrained policies are uncommon for simple classic-control tasks, while we used Go2 to evaluate high-level policy-guided locomotion rather than repeating similar experiments on G1. We will add these selection reasons and the resulting limitations to Appendix A.