acceptodds
Withdrawn from ICLR 2027

Transferring the Intelligence of VLMs to Robotic Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

34 views

Abstract

Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control? We investigate this question through RoboDawn, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands. Using this interface, the VLM controls a robot in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions to the resulting state. Furthermore, we introduce an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies. Experiments on RoboTwin 2.0 C2R and RoboDojo demonstrate that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, RoboDawn outperforms several strong policies trained on benchmarkspecific robot data, while a single in-context demonstration further yields substantial performance gains and establishes state-of-the-art (SOTA) results. On RoboTwin 2.0 C2R, the success rate increases from 53.2% zero-shot to 73.6% one-shot, exceeding the solid baseline π0.5 (46.0%). Similar gains are observed on RoboDojo, where success rate improves from 35.67% zero-shot to 47.17% one-shot. The same framework also transfers to real-world robots, performing block-in-basket and block stacking on Franka.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Models, Zero-Shot Generalization, General Robot Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GXioOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RoboDawn, a framework that enables frozen VLMs to perform closed-loop robotic manipulation through semantic commands for translation, rotation, and gripper control. Its in-context learning scheme combines a shared command primer with task demonstrations converted into the same action interface and augmented with observations, execution effects, and rationales. The paper reports success-rate improvements from 53.2% to 73.6% on RoboTwin 2.0 C2R and from 35.67% to 47.17% on RoboDojo when adding one task demonstration. Ablations examine demonstration count, VLM choice, and harness components. Preliminary real-world evaluation achieves 9/10 successes for block-in-basket, 5/10 for stacking, and 0/10 for cloth folding.

Critical strengths
  1. Substantial empirical gains with very limited task demonstrations. On RoboTwin, one-shot RoboDawn reaches 73.6%, compared with 53.2% for its zero-shot variant and 58.4% for the strongest listed baseline. The improvement also appears on RoboDojo, where success increases by 11.5 percentage points. These results suggest that demonstrations can provide useful adaptation to randomized manipulation scenes without task-specific parameter updates.
  2. A coherent alignment between demonstrations and executable actions. Expressing expert trajectories in the model’s command vocabulary, and consistently using the gripper interaction point for observations and control, provides a practical way to reduce mismatches between demonstration context and execution.
Critical weaknesses
  1. The contribution of the specific ICL design is not isolated. The experiments show that adding a demonstration helps, but do not compare the proposed format with simpler guidance derived from the same trajectory, such as a textual task plan. Thus, the benefit attributable to interface-aligned ICL remains unclear.
Questions for authors
  1. What does “5 runs” mean in terms of episodes and scene seeds per task? How do RoboDawn’s command limits translate into physical execution horizons, and were the compared policies evaluated under equivalent termination rules?
  2. Where do the one-shot demonstrations for the eight Open tasks come from? How is the demonstration selected for each task, and is performance sensitive to that selection? What robot observations and examples are included in the shared primer that remains present in the zero-shot setting?
  3. Can you provide controlled evidence, using the same VLM and task demonstration, that the complete proposed format improves over simpler alternatives such as a demonstration-derived textual plan or command trajectory without images and rationales?
Reviewer_F9ZbOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RoboDawn, a harness framework designed to enable frozen VLMs to control robots without task-specific training. The framework mainly incorporates execution feedback, memory, and in-context demonstrations into pretrained VLMs. Given camera images and robot states, the model reasons about the next action and issues discrete movement, rotation, and gripper commands in a closed loop. Expert trajectories are converted into demonstrations compatible with this action interface. The design improves performance over several robot-trained policies on RoboTwin 2.0 C2R and RoboDojo. With GPT-6 Astra, one demonstration increases success rates from 53.2% to 73.6% on RoboTwin and from 35.67% to 47.17% on RoboDojo. Real-robot evaluations with Gemini include block placement, stacking, and cloth folding, with mixed results.

Critical strengths
  1. The paper shows promising robot-control results without task-specific training.

  2. The paper evaluates multiple VLMs and provides useful ablations of the harness components.

Critical weaknesses
  1. The method mainly combines a frozen VLM with an action interface, memory, and demonstrations. The technical novelty beyond existing robot harnesses is limited.
  2. The main results use Astra, but the harness ablations use Gemini. The contribution of each component to Astra’s performance remains unclear.
  3. Real-world evaluation includes only three tasks with ten trials each, providing limited support for the broader claims of intelligence transfer.
Questions for authors
  1. What is technically new compared with existing methods such as Show-Harness, particularly in the demonstration design?
  2. Was the original Astra baseline evaluated with the same observations, controller, and inference budget as RoboDawn?
  3. Are harness ablation results available for Astra under the main one-shot setting?
Reviewer_9bs1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboDawn enables a frozen VLM to control robots through semantic translation, rotation and gripper commands, with multi-view observations, robot state, execution feedback and interaction memory. Expert trajectories are converted into interface-compatible in-context demonstrations containing observations, commands, action effects and post-hoc rationales. On RoboTwin 2.0 C2R, GPT-6 Astra achieves 53.2% success zero-shot and 73.6% one-shot, compared with 46.0% for π0.5. On RoboDojo, success increases from 35.67% to 47.17%. Preliminary real-robot experiments use Gemini 3.8 Flash.

Critical strengths
  1. The paper addresses a relevant question: how effectively can existing VLM capabilities transfer to robot control without task-specific parameter updates? The results across two simulation benchmarks are promising.
  2. A single task demonstration produces substantial gains: +20.4 percentage points on RoboTwin and +11.5 on RoboDojo. Converting existing expert trajectories into the online command interface is a practical way to make robot demonstrations usable by frozen VLMs.
  3. The paper provides shot-count, backbone and harness ablations, as well as command-budget analysis and preliminary real-world results. It explicitly acknowledges limitations in speed and fine-grained manipulation.
Critical weaknesses
  1. The central ICL contribution is not sufficiently isolated. The zero-shot versus one-shot results convincingly demonstrate that task examples improve the complete system. However, they do not establish whether the proposed demonstration construction offers an advantage over simpler presentations of the same expert trajectory, such as keyframes with action descriptions. Since a same-task demonstration provides substantial information about the solution strategy, the gains could arise partly from this task guidance rather than the specific interface-aligned representation. A focused comparison using the same source trajectory and comparable information content would help distinguish these explanations.

  2. The existing ablations do not identify which aspects of the demonstration representation matter. Table 3(c) evaluates reasoning, grids, and the command primer in the zero-shot setting. These experiments support the harness design, but do not separately assess the post-hoc rationales or explicit action effects included in task demonstrations. Consequently, it remains unclear which components are necessary for the reported one-shot gains. The C2R evaluation does demonstrate scene generalization; however, it does not distinguish task-specific guidance from learning reusable interface knowledge. This limits the interpretation of the ICL gains, rather than establishing that the system lacks generalization. A simpler demonstration baseline and a small number of targeted ablations would be the most direct way to strengthen the central claim.

  3. The contribution should be positioned more precisely relative to closely related work. Show-Harness also investigates frozen VLM control through semantic action interfaces and explores video-conditioned task execution (§5.3.2). Its demonstration experiment concerns following an object order, rather than RoboDawn's interface-aligned action trajectories, so the approaches are not equivalent. Given that Show-Harness is concurrent work, I do not consider a direct experimental comparison essential. Nevertheless, a clearer explanation of what is new in the demonstration construction—and what evidence supports that distinction—would strengthen the paper beyond the broader claim of transferring VLM intelligence to robot control.

Questions for authors
  1. How does the proposed demonstration representation compare with a simpler presentation of the same expert trajectory, such as keyframes with action descriptions?
  2. What are the separate contributions of post-hoc demonstration rationales and explicit action effects? Would the gains largely remain with observations, robot states, and commands alone?
  3. Can the authors clarify whether the main contribution is the demonstration conversion procedure, its structured presentation, or the empirical finding that one-shot conditioning substantially improves frozen-VLM robot control?

The paper does not compare its structured demonstrations with a simpler presentation of the same expert trajectory. It therefore remains unclear whether the gains come from the proposed representation or simply from showing the VLM how to solve the task. This unresolved attribution is my main reason for recommending weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.