acceptodds
Under review as a conference paper at ICLR 2027

Scaffolding General Vision Language Models for Zero-Shot Robot Control

Abstract

Vision-language-action (VLA) models have advanced robotic manipulation, but their zero-shot generalization remains limited, and their reliance on specialized robotics data keeps them from benefiting directly from rapidly advancing general-purpose vision-language models (VLMs). Agentic robotic systems instead use VLMs for high-level reasoning, yet typically depend on substantial external modules. This motivates our question: Can a general-purpose VLM itself serve as the decision-making core for robotic manipulation? Evaluating VLMs on action selection, progress assessment, and task completion reveals a capability-control gap: VLMs reason well about manipulation, but their decisions do not match the abstraction of continuous, embodiment-specific control. We introduce MotorMind, a VLM-centric manipulation harness with an asynchronous inference loop. Without task-specific policy training, MotorMind achieves 66.7% base success and 53.8% success under perturbations on LIBERO-PRO, compared with at most 13.3% and 19.2% for the prior zero-shot methods we evaluate, and the same interface reaches 95% average success on a real xArm6 robot across direct manipulation and human-perturbation settings. A stronger VLM backbone further raises base success without changing the harness, and the remaining errors, mainly in visual grounding, embodied reasoning, and action knowledge, decrease with stronger VLMs. These results reframe zero-shot manipulation as aligning general-purpose model capabilities with the control representation, rather than solely learning specialized action policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Models, Robot Manipulation, Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X1xUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MotorMind, a framework that uses a frozen general-purpose VLM for zero-shot robotic manipulation. It assigns the VLM multiple roles for planning, action generation, monitoring, verification, and memory, and connects these roles to a deterministic controller through mid-level actions and asynchronous execution. The paper evaluates the framework on LIBERO-PRO, adaptive manipulation tasks, and a real xArm6 robot, reporting task success and execution efficiency.

Critical strengths
  1. The paper presents a complete closed-loop manipulation system without task-specific policy training. Its ability to revise actions based on execution feedback, together with the real-robot experiments under human perturbations, provides useful evidence that general-purpose VLMs can support physical manipulation.

  2. The evaluation considers execution time alongside success rate. This is consequential for a framework involving repeated VLM calls, as it makes the trade-off between manipulation performance and inference overhead more explicit.

Critical weaknesses
  1. The physical execution process is insufficiently specified. The paper describes mid-level actions and controller checks, but does not clearly explain how visual observations are translated into robot-frame motions or how target distance and clearance are obtained. This makes it difficult to reproduce the system and assess how much of its performance comes from the VLM versus the controller and its associated rules.

  2. The near-zero performance of most direct zero-shot VLA baselines limits the conclusions supported by these comparisons. Some configurations involve embodiment transfer, so their failures may reflect control-interface or input-distribution mismatch rather than task-level generalization alone. Although the paper also evaluates agentic baselines, a matched comparison using the same VLM and controller would provide clearer evidence for the proposed harness.

Questions for authors
  1. Execution clarification: How are visual observations and mid-level actions converted into robot-frame motions? What information is used for target-distance and clearance checks, and which parts require calibration, geometric estimation, or manually specified rules?

  2. Baseline clarification: How did you verify that the near-zero VLA results are not primarily caused by embodiment, action-normalization, or control-interface mismatch? Which evaluated configurations support a task-generalization comparison independently of these factors?

  3. Controlled comparison: Have you evaluated a simpler synchronous closed-loop agent using the same VLM, observations, controller, and execution budget? Such a comparison would materially affect my assessment of whether the proposed multi-role asynchronous design is responsible for the gains.

6 comments
AuthorsOpenReview ↗

Response to Reviewer X1xU

We thank the reviewer for the precise questions and for recognizing the closed-loop system and the efficiency evaluation. We answer each question below.


1. More details on the execution process of MotorMind

We have updated the PDF with more detail (Section 3; Appendix C, Table 8, Fig. 8; Appendix D,; Appendix J)

In short, each control cycle has three steps:

  1. Locate. The same frozen VLM returns 2D boxes for the objects in each camera view and names the target.
  2. Express in the robot frame. The target box is back-projected with the camera, cross-checked across views and refined from the wrist view.
  3. Act. The Executor receives the images, the robot state and the tool-to-target offset, and writes up to four actions (move, rotate, gripper, home, wait). The controller converts them into base-frame Cartesian or gripper commands.

We also provide more details on the three ingredients the reviewer asks about:

  • Calibration. Camera intrinsics and camera-to-base extrinsics, the table height, and the gripper's open span. These are all real-robot-equivalent information.
  • Geometric estimation. Converting the VLM's boxes into base-frame offsets, and reading clearance from wrist depth.
  • Verification. Execution checks: reach, a floor margin above the table, close depth, release position, and turn and tilt limits.

Everything else is decided by the VLM: task decomposition, target localization, action selection, monitoring and completion judgment are all calls to the same model. No learned detector, segmentation model, IK solver or motion planner is used.

We also add the VLM versus controller results in Table 11: In 18,241 proposed actions, 68.4% are executed, 29.0% are rejected before execution, and 2.6% are not reached. A rejection ends the batch, and its reason goes back to the VLM, which writes the next proposal. The checks bound and filter individual motions; they never choose the target object or the plan. The prompts for every role are in Appendix J.


2. Zero-shot VLA baselines

We’d like to clarify that the near-zero results of zero-shot VLA exactly show the generalization limitation of VLA.

With the same evaluation code, observation transforms and action conversion, the LIBERO-adapted checkpoints reach high success: 98.3% (π0.5), 100.0% (MolmoAct2) and 98.3% (OpenVLA-OFT). Each zero-shot model runs with its own released preprocessing, action decoding and normalization, listed per model in Appendix E.2.

We further analyze the rollouts of each zero-shot VLA results. The base checkpoints do not produce coherent, target-directed motion on the LIBERO arm. This points to an embodiment and action-distribution mismatch, not a task-level error. It is the limitation that motivates our approach: a VLA policy needs embodiment-specific adaptation before it can act, whereas MotorMind's VLM-facing interface is unchanged between the simulated Franka and the real xArm6.

In addition, we provide the comparisons independent of embodiment mismatch. The LIBERO-adapted VLAs run on their own embodiment, so their perturbation results isolate task-level generalization:

MethodSemanticObjectPositionTask
π0.5 (LIBERO-adapted)98.395.036.723.3
MolmoAct2 (LIBERO-adapted)95.090.036.730.0
OpenVLA-OFT (LIBERO-adapted)96.786.711.710.0
MotorMind (zero-shot)58.346.751.758.3

Adapted VLAs lead when the task matches their training distribution. When object positions or the task itself change, they fall to 10–37%, and MotorMind leads at 51.7% and 58.3% with no task-specific training. The agentic baseline CaP-X does not depend on VLA transfer and uses stronger models, yet it reaches 13.3% base and 19.2% under perturbation, against 66.7% and 53.8% for MotorMind.


3. A synchronous agent with the same VLM, observations, controller and budget

We additionally add a Serial execution ablation. It keeps the same VLM, observations, prompts, action interface, controller and budget, and removes only concurrency: monitoring and memory updates run between steps. Results on 60 paired episodes, with the full system re-run for this comparison:

ConfigurationSuccess (%)Avg. time (s)
MotorMind (full)66.7217
Serial (synchronous)45.0297
6D pose actions40.0253
2D point actions18.3312
Without replanning36.7—
Without verifier60.0—

The asynchronous design accounts for a 21.7-point gain and saves 80 s per episode. The mid-level action interface accounts for 26.7 points over 6D pose and 48.4 points over 2D points. Within the roles, removing replanning lowers success to 36.7%, and removing the Verifier lowers it to 60.0%.

As the discussion period draws to the close soon, we would greatly appreciate any further feedback or follow-up you may have regarding our rebuttal. If our response has satisfactorily addressed your major concerns, we would also be grateful if you could reconsider our merit and update the rating.

We thanks again for your evaluation, suggestion and your time! Have a wonderful weekend.

AuthorsOpenReview ↗

Following up on our response

Hi Reviewer X1xu,

We sincerely thank the reviewer for their feedback and suggestions! We have carefully addressed all three concerns through additional clarifications and experiments:

  1. Execution transparency: We expanded the implementation details, including calibration, geometric estimation, controller checks, and the respective responsibilities of the VLM and controller.
  2. Baseline validity: We clarified the impact of embodiment mismatch of VLA evaluation, and provided additional comparisons to better assess task-level generalization.
  3. Controlled ablations: We conducted matched experiments using the same VLM, observations, controller, and budget, demonstrating an improvement from 45.0% to 66.7% success over the synchronous baseline, alongside further component ablations.

We hope these revisions and new results have addressed the reviewer's concerns and strengthened the paper. We would be very grateful if the reviewer could kindly revisit their assessment once they got chance. We truly appreciate the reviewer's time, consideration, and valuable feedback throughout the review process.

Reviewer_6CbNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MotorMind, a zero-shot robotic manipulation framework that eliminates the need for task-specific policy training. The system leverages a single frozen Vision-Language Model to handle planning, action generation, monitoring, verification, and memory. It bridges semantic reasoning and robot control via mid-level action representations, while utilizing asynchronous feedback to facilitate online error correction and replanning. On the LIBERO-PRO benchmark, the framework achieved average success rates of 66.7% and 53.8%, respectively. Furthermore, on a real-world xArm6 platform, it attained a combined success rate of 95% across direct-perception and manually-perturbed placement tasks, demonstrating the feasibility of the approach.

Critical strengths
  1. MotorMind achieves a high success rate among the zero-shot baselines evaluated, without requiring demonstrations from the LIBERO dataset or task-specific policy fine-tuning. The results obtained on a xArm6 robot provide real-world valuable evidence for the practical viability of this approach.
  2. The framework employs a single frozen VLM to unify planning, action generation, monitoring, verification, and memory, with a deterministic controller executing the actions, thereby obviating the need for additional action-policy training
Critical weaknesses
  1. The proposed framework shares substantial similarities with existing agentic manipulation systems, particularly Harness VLA and CaP-X, in hierarchical planning, feedback-driven execution, verification, and recovery. Its main distinctions appear to lie in replacing learned or code-based action generation with direct mid-level VLM commands and introducing asynchronous monitoring. However, the current experiments do not sufficiently isolate the contributions of these design choices from differences in action representations, underlying models, and available control tools.

  2. The existing analyses characterize what failures occur, but provide limited insight into which failures the system can recover from, how decision-level errors translate into physical outcomes, and how these patterns change with task complexity.

  3. Real-world experiments focus primarily on pick-and-place tasks, with variations mainly limited to target descriptions and external disturbances; thus, the generalization of the approach to a broader range of manipulation skills in real-world remains insufficiently validated.

  4. Although asynchronous execution is highlighted as a key contribution, the paper does not provide a controlled ablation isolating its impact. The authors use the sequential VoLo framework as a proxy baseline (Appendix D), but VoLo differs from MotorMind in perception, action generation, memory, and execution mechanisms. A synchronous variant of MotorMind with all other components fixed, together with measurements of inference and execution overlap, would be necessary to substantiate the claimed efficiency gains.

Questions for authors
  1. Does the quality of subsequent actions within a short batch degrade as the prediction horizon extends or as execution deviations accumulate?

  2. In Table 7, the accuracy for rotation action selection by Qwen3.8 is 0, and for translation, it is only 48% Given these figures, how is an overall success rate of 66.7% achieved for the closed-loop baseline tasks?

  3. The Table 15 shows non-monotonic success rates, dropping from 66.7% at 300 ~ 450s to 50.0% at 900s. Although the authors acknowledge this observation, its underlying causes remain unclear. This raises concerns about the robustness of the proposed execution and recovery design, as allowing more time for task completion appears to degrade performance rather than improve it.

2 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer 6CbN

We thank the reviewer for the detailed questions and for recognizing the zero-shot results and the real-robot evidence. We respond to each point below.


1. What distinguishes MotorMind, and isolating those choices

First, we agree planning, feedback and verification are indeed common to other agentic systems, however, MotorMind differs in two design choices, which make the MotorMind unique. The first is how actions are produced: the VLM writes mid-level actions directly, with no generated code and no learned action policy. The second is how reasoning is scheduled: monitoring and memory updates run concurrently with motion.

Second, we’d also like to clarify that Harness VLA and VoLo use the same Qwen3.8-Flash-Next planner as MotorMind, and CaP-X uses stronger models (GPT-5.6-Terra and Claude Opus 5) yet reaches 13.3% base success against our 66.7%. Hence, the backbone capability does not explain our results.

To isolate the two design choices, we ran additional ablations on 60 episodes with the VLM, observations, controller and budget fixed. The full system was re-run for this comparison, so its per-suite counts in Table 22 differ from Table 3 while the total is the same (40 of 60).

ConfigurationSuccess (%)Avg. time (s)
MotorMind (full)66.7217
Serial (no asynchrony)45.0297
6D pose actions40.0253
2D point actions18.3312

The results show that both choices matter. Replacing the mid-level interface costs 26.7 points with 6D poses (p = 0.006) and 48.4 points with 2D points. Removing asynchrony costs 21.7 points (p = 0.027).


2. Controlled asynchrony ablation

The serial baseline in point 1 above is the synchronous variant the reviewer asks for. Monitoring and memory updates run between steps instead of concurrently, and every other component is unchanged. Success falls from 66.7% to 45.0%, uniformly across suites (9 of 20 episodes in each), and mean episode time rises by 80 s (36.9%). Both variants make the same calls, so the 80 s is the inference time that the full system overlaps with motion.


3. Recoverable failures, physical outcomes and task complexity

The table lists how many of the affected episodes still succeed. An episode can contain more than one failure type, so the last column counts each episode once, by its first failure.

Failure typeEpisodesEventually successfulRateAs first failure: episodes / successful (rate)
Motion blocked / stuck1697846.2%138 / 65 (47%)
Failed grasp341338.2%22 / 12 (55%)
Incorrect target / grounding25624.0%7 / 2 (29%)
Misplacement / dropped object641421.9%24 / 9 (38%)
Incorrect completion judgment4748.5%7 / 1 (14%)

To answer reviewer question, we clarify following points in the paper:

  • Which failures are recoverable. About half of the episodes that begin with a blocked motion or a failed grasp still succeed. Completion and grounding errors are harder to recover from, and they mostly arise after an earlier failure: 47 episodes contain a completion error, but only 7 begin with one.
  • From decision errors to physical outcomes. A grounding error appears physically as a wrong-object or empty grasp, and a premature completion judgment as a false-done ending ( we added failure-analysis figures in Section 6 and Appendix I).
  • Task complexity. Pooling the two independent runs of the full system (Table 3 and Table 22), success is 75.0% on Object, 70.0% on Spatial, and 55.0% on Goal. Harder conditions also change the failure pattern. Under perturbation, the share of episodes with a physical failure rises from 43% to 56%, and the share of failing episodes that recover falls from 23%to 18%( we added failure-flow figure, Appendix I).
AuthorsOpenReview ↗

[2/2] Response to Reviewer 6CbN

4. Action quality within a batch

Thanks reviewer for this interesting question. We further break action quality down by position in the batch.

For each position it gives the number of proposed actions that were judged, the share that passed validation, and the share of those that were executed unchanged.

Batch positionJudged actionsValid rateUnchanged among valid
17,19685.9%70.8%
25,61766.7%86.1%
33,33960.4%60.7%
41,61933.5%85.2%

We have two additional insights from this analysis. First, execution error does not accumulate: for actions executed unchanged, the median end-effector deviation is 3 mm at every position. Second, the validity of later proposals falls largely because of what those actions are: position 3 is often a descent that stops on contact, and position 4 a gripper close that depends on the actions before it. This is why batches are short, every action after the first is re-checked against a fresh observation, and the Monitor can cancel the rest of a batch.


5. Rotation accuracy of 0% versus 66.7% closed-loop success

First, we’d like to clarify that the two numbers measure different things. The diagnostic asks the VLM to pick the next primitive from 14 options, given two still frames, no robot state and no feedback. In the closed loop, the Executor infer a direction with the help of other components in Motormind: It receives the target offset as base-frame direction words, computed from its own localization of the target. It also sees the robot state, which told how far each move or turn actually went, and the VLM will correct a wrong one in its next proposal.

Second, the replanning ablation shows how much this feedback contributes: success falls from 66.7% to 36.7% without it.

Lastly, on the real robot, the full system completes the 180° rotation task in 10 of 10 trials. Motormind has this capability.


6. Non-monotonic success across budgets

We’d like to clarify that the results do not show that more time degrades performance. Each row of Table 21 is a separate set of stochastic rollouts over 30 episodes, not the same trajectories given a longer deadline. The gap between 450 s and 900 s is 20 versus 15 successes out of 30 (Fisher exact p = 0.29), which is within sampling variation.


7. Real-robot tasks beyond pick-and-place

We extended the real-robot evaluation with three tasks beyond pick-and-place (pushing, stacking and a 180° rotation) and two agentic baselines, CaP-X and ASPIRE, both running on GPT-6 Astra. Success rates (%) over 10 trials per task:

MethodCube to BowlBattery into BoxPushingStackingRotation (180°)
CaP-X000100
ASPIRE8080010020
MotorMind100100100100100

MotorMind also uses fewer tokens. Average total tokens per successful execution (ASPIRE's exploration tokens are not counted):

TaskMotorMindASPIRE
Stacking332.6k844.8k
Pushing59.5kno success
Rotation (180°)150.7k376.7k

ASPIRE uses 2.54× and 2.50× the tokens of MotorMind on stacking and rotation.

The baseline failures are systematic and come from the external modules the baselines depend on. CaP-X scores grasps against the Panda finger model, and its grasps end with the xArm fingertips below the table. ASPIRE's motion planner rejects goals that are in fact clear.


We sincerely thanks again for the reviewer for the careful assessment and suggestions.

Reviewer_FfHXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MotorMind, a VLM-centric framework for zero-shot robotic manipulation. A single frozen general-purpose VLM serves as planner, executor, monitor, verifier, and memory manager. It generates short batches of parameterized end-effector translation, rotation, and gripper actions, which a deterministic controller validates and executes. Asynchronous monitoring and replanning support correction during execution. Using Qwen3.8-Flash-Next, the system reports 66.7% success on the LIBERO-PRO base tasks and 53.8% under perturbations, outperforming the zero-shot baselines evaluated in the paper. The authors also demonstrate tabletop manipulation on an xArm6 and report improved base-task performance with GPT-6 Sol in a single-seed backbone comparison.

Critical strengths
  • The framework provides a coherent decomposition of manipulation into complementary reasoning roles. Planning, action generation, monitoring, verification, and memory have distinct responsibilities, and the action interface connects these decisions to physical execution. The recovery example and the replanning ablation provide evidence that this organization can help compensate for imperfect individual decisions. This is a useful system design for applying general-purpose VLMs to robotic manipulation without task-specific policy training.

  • The reported zero-shot results provide encouraging evidence for this approach. MotorMind substantially outperforms the evaluated zero-shot baselines on both the base and perturbed LIBERO-PRO settings. The real-robot demonstrations establish feasibility on the tested tabletop tasks. The improvement from 66.7% to 83.3% in the separate GPT-6 Sol comparison also suggests that the framework can benefit from a stronger backbone, although this evidence is limited to a single seed and comes with increased execution time.

Critical weaknesses
  • The connection between the capability diagnosis and the claimed mechanism of improvement is insufficiently established. The Qwen backbone obtains 36.25% action-selection accuracy, 55% progress-assessment accuracy, and 65% completion accuracy in the offline diagnostic, while the full system reaches 66.7% task success. These figures are not directly contradictory: the evaluations differ in inputs, state distributions, action formulation, recovery opportunities, and success criteria. However, the paper does not sufficiently establish how the proposed design turns these unreliable local decisions into successful execution. The replanning and verifier ablations, together with the recovery analysis, provide partial evidence, but do not isolate the effects of the mid-level action representation, asynchronous scheduling, and controller-side corrections. The controller can reject or clip commands based on geometric and gripper-state checks, while the appendix additionally mentions geometric action corrections without adequately specifying their implementation or information sources. The PDF provides only abbreviated prompts. These omissions make it difficult to assess which aspects of performance come from VLM action generation and which depend on the surrounding execution logic. Moreover, Appendix G states that simulation episodes stop when the environment's goal predicate becomes true, so reaching the goal does not by itself establish reliable autonomous completion recognition. The evidence therefore supports the effectiveness of the complete system more clearly than its explanation of how the capability–control gap is addressed.
  • The real-robot evaluation has limited task diversity and comparative evidence. The reported 95% success rate comes from 120 trials across three objects and two receptacles, evaluated with direct instructions and human perturbations. This is useful repeated testing, but the physical operations remain concentrated on tabletop pick-and-place. The additional semantic tasks primarily vary target interpretation rather than the physical demands of manipulation. Without real-robot baselines or evidence covering different manipulation requirements, these results support feasibility in the tested setting more strongly than broad real-world manipulation capability. This is a limitation of coverage and interpretation, rather than simply an insufficient number of trials.
Questions for authors
  • Could the authors specify the complete action-generation and execution interface, including the Executor's structured outputs, the source of translation and rotation magnitudes, and the information used by the controller for target-distance, clearance, and gripper checks? Please clarify what the appendix's geometric action corrections do, how often proposed actions are rejected or modified, and which decisions remain attributable to the VLM. Complete prompts and representative traces showing proposed, corrected, and executed actions would help resolve this concern.

  • What evidence connects the diagnostic weaknesses to the mechanisms responsible for closed-loop success? The reported replanning ablation and recovery analysis are helpful; an analysis linking incorrect decisions, controller intervention, retries, and eventual recovery, or a targeted comparison holding inputs and controller behavior fixed, would make the explanation more convincing. Please also distinguish reaching the environment-defined goal from the VLM correctly recognizing completion and stopping, given the predicate-based termination described in Appendix G.

2 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer FfHX

We thank the reviewer for the careful reading and for recognizing the system design and the zero-shot results. We respond to each concern below.


1. How imperfect local decisions become closed-loop success

First of all, we appreciate the reviewer notices this central question, which exactly highlights the core contribution of MotorMind. MotorMind’s design is the key to bring VLM to success in close-loop evaluation. To show how much each mechanism contributes, we ran additional ablations: Each variant changes one mechanism and keeps the VLM, observations, controller and budget fixed. The full system was re-run for this comparison.

ConfigurationSuccess (%)Avg. time (s)
MotorMind (full)66.7217
Serial (no asynchrony)45.0297
6D pose actions40.0253
2D point actions18.3312
Without replanning36.7—
Without verifier60.0—

Second, we’d also like to clarify that the two evaluation settings are different. The diagnostic scores a single decision from two frames with no feedback. In closed loop, the same VLM acts through three mechanisms that the diagnostic does not have: a mid-level action interface, asynchronous monitoring, and feedback-driven replanning.

Third, we'd like to clarify on isolating the controller. The controller is a translation step: it converts the VLM's mid-level actions into Cartesian and gripper commands. It cannot be removed on its own, because without it no action reaches the robot. Its contribution is therefore measured together with the interface it serves: the 6D-pose and 2D-point variants replace both the interface and its translation step, and they lose 26.7 and 48.4 points. The validation checks applied to each action before it runs only constrain execution. Point 2 reports how often they act (Table 11).

Finally, a wrong decision does not have to end the episode, because the system detects failures and recovers from them. The table lists how many of the affected episodes still end in success. These counts include every detected failure event, including transient ones.

Failure typeEpisodesEventually successfulRate
Motion blocked / stuck1697846.2%
Failed grasp341338.2%
Incorrect target / grounding25624.0%
Misplacement / dropped object641421.9%
Incorrect completion judgment4748.5%

2. The execution interface and what is attributable to the VLM

In short, every key decision is made by the VLM. The rest of harness converts its output into robot coordinates and checks each motion during it runs.

To further clarify this, we revised the paper to specify each part:

  • Interface. Each Executor reply is one JSON object with five fields: assessment, done, command, expect and confidence. The command holds up to four actions. Each action is a move (a direction or base-frame axis, and a distance in mm), a rotation (a direction and an angle in degrees), a gripper command.
  • Division of workload. The VLM performs task decomposition, target localization, action selection, monitoring, completion judgment and replanning. Geometry only re-expresses the VLM's own localization in the robot frame, and the resulting offset is passed back to the Executor, which writes the motion. No learned detector, segmentation model, IK solver or motion planner is involved.
  • Information used . Target distance comes from the VLM's boxes and the camera calibration, clearance from the wrist depth camera, and gripper checks from the measured finger opening.
  • Execution checks. Before an action runs, it is checked for reach, a floor margin above the table, close depth, release position, and turn and tilt limits. Every action after the first in a batch is re-checked against a fresh observation.
  • How often the checks act. Of 18,241 proposed actions (batch positions 1 to 4), 68.4% are executed, 29.0% are rejected before execution, and 2.6% are not reached. A rejection ends the batch, and its reason is returned to the VLM, which writes the next proposal. The checks bound and filter individual motions; they never choose the target object or the plan.
  • Prompts. The prompt for every role is included.

To further address the reviewer's concern, we illustrate traces of one subgoal through the execution loop in Fig. 8. It shows where a proposal is parsed, bounded, edited, gated against a fresh observation, executed and measured, and where a refusal returns to the Executor.

AuthorsOpenReview ↗

[2/2] Response to Reviewer FfHX

3. Reaching the goal versus recognizing completion

First, we’d like to clarify we follow the standard LIBERO-Pro evaluation protocol to evaluate all methods. The automatic end is the standard protocol, since VLA baselines are not able to auto stop.

Second, to answer the reviewer's question on whether success depends on the environment ending the episode once the goal predicate holds, we add additional experiments to prove it does not. We re-ran the evaluation with predicate-based stopping disabled, so the system must decide for itself when the task is complete. The predicate is read once, when the system ends the mission itself, and is never shown to any model.

SettingWith env. stopWithout env. stop
Base success66.7%65.0%
Perturbation success53.8%57.1%
Avg. time243.5 s274.0 s

Success is only slightly affected within run-to-run variation, so the results do not depend on the environment ending the episode. MotorMind has ability to stop the session when task is success. This is also shown in the real robot experiment, where we don't have environment early stop.


4. Real-robot diversity and baselines

We extended the real-robot evaluation with three tasks beyond pick-and-place (pushing, stacking and a 180° rotation) and two agentic baselines, CaP-X and ASPIRE, both running on GPT-6 Astra. Success rates (%) over 10 trials per task:

MethodCube to BowlBattery into BoxPushingStackingRotation (180°)
CaP-X000100
ASPIRE8080010020
MotorMind100100100100100

MotorMind also uses fewer tokens. Average total tokens per successful execution (ASPIRE's exploration tokens are not counted):

TaskMotorMindASPIRE
Stacking332.6k844.8k
Pushing59.5kno success
Rotation (180°)150.7k376.7k

ASPIRE uses 2.54× and 2.50× the tokens of MotorMind on stacking and rotation.

The baseline failures are systematic and come from the external modules the baselines depend on. CaP-X scores grasps against the Panda finger model. The xArm gripper's fingertips reach lower and sweep deeper as they close, so the selected grasps place the fingertips 0–6 mm below the table and trigger a collision fault. ASPIRE's motion planner rejects goals that are in fact clear, because its collision world treats the manipulated object as an obstacle and models the fingers as always open. It further demonstrates the advantage of MotorMind having no grasp model or motion planner to adapt.


We sincerely thanks again for the reviewer for the careful assessment and suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.