acceptodds
Under review as a conference paper at ICLR 2027

VLMs are Native Action Models: Unleashing the manipulation capabilities of VLMs with Explicit Action-Space Grounding

Abstract

Pretrained vision-language models (VLMs) already exhibit strong visual, spatial, and reasoning capabilities relevant to robotic manipulation. A key challenge in turning these capabilities into generalist manipulation policies is how to expose these capabilities through an appropriate action interface. Vision-language-action (VLA) models provide a unified trainable policy, but typically introduce learned action tokens, action heads, or action experts whose semantics must be acquired from robot trajectories before the model can act. In contrast, modular foundation-model-based manipulation methods can exploit pretrained VLM capabilities for zero-shot manipulation, but rely on external planning and control scaffolds that are not end-to-end trainable as a unified continuous-action policy. We introduce EASG, an Explicit Action-Space Grounding framework that bridges these two paradigms. EASG makes the semantics of a continuous robot action space directly observable in the VLM’s perceptual context by projecting a tool center point (TCP)-centered action frame into first-person observations, while expressing actions through the VLM’s native textual generation interface. This allows pretrained VLMs to directly translate their existing visual-spatial reasoning into executable robot actions, without robot-action pretraining or auxiliary learned modules. At the same time, the same interface supports direct fine-tuning with robot trajectories, allowing the model to continuously improve upon the manipulation capabilities unleashed by EASG. We demonstrate that EASG enables zero-shot manipulation and generalization across object positions, categories, and robot embodiments, while remaining effectively trainable in continuous action spaces. These results suggest that, given an appropriate action interface, pretrained VLMs can themselves serve as native action models for robotic manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Artificial Intelligence, Vision Language Model, Vision Language Action Model, Code as Policy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aF7gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Explicit Action-Space Grounding (EASG), a framework that allows pretrained vision-language models (VLMs) to produce executable robot actions without any robot-action pretraining. EASG overlays a tool-center-point (TCP) coordinate frame onto wrist-camera observations, supplies explicit metric depth information, and describes the action semantics through language instructions. Using its original textual generation interface, the VLM outputs translation, rotation, gripper, and termination commands, while robot-specific execution is handled by deterministic coordinate transforms and inverse kinematics. The same action interface can subsequently be fine-tuned on robot trajectories, with grammar-constrained decoding enhancing the reliability of generated outputs.

The authors assess EASG via real-world zero-shot manipulation, robustness to object and pose changes, fine-tuning on more difficult manipulation tasks, LIBERO benchmark evaluations, and cross-embodiment transfer. The findings indicate that, given a suitable action interface, pretrained VLMs can exhibit non-trivial manipulation ability and can improve further through robot-data fine-tuning.

Critical strengths
  • Convincing Zero-Shot Manipulation from a Pretrained VLM. EASG enables a pretrained VLM to reach a 79.4% average zero-shot success rate on six real-world tasks, on par with VLA baselines fine-tuned with task-specific demonstrations, and it remains comparatively stable under object and pose randomization. This suggests that capabilities usually acquired through robot-action training can be elicited through an explicitly grounded interface.
  • A Unified, Trainable Action Interface. The same textual action interface supports both zero-shot execution and robot-data fine-tuning without learned action heads or action tokens, yielding 67.2% on six harder real-world tasks, 83.3% on LIBERO with a 0.8B VLM, and direct UR3-to-Flexiv transfer with only a small drop. This supports the promise of explicitly grounded, embodiment-independent action representations.
Critical weaknesses
  • Insufficient Isolation of the Grounding Contribution. The comparisons do not clearly control for differences in sensory information (RGB-D metric depth), model capacity, and inference-time computation: Table 7 shows zero-shot success plunges from 79.4% to 11.7% without thinking, indicating that inference-time reasoning accounts for much of the reported performance. More carefully controlled comparisons are needed to separate the contributions of the grounding interface, reasoning, and extra geometric information.
  • Missing Runtime Analysis and Limited Generalization Scope. The paper reports no action-prediction latency, control frequency, reasoning-token overhead, or task completion time, which is especially consequential given the strong dependence on thinking and makes practicality hard to judge. Moreover, the cross-embodiment results share the same gripper, matched initial poses, and similar workspaces, so they mainly demonstrate transfer across arms rather than broader generalization to different end-effectors or environments.
Questions for authors
  • Inference latency and reasoning overhead. Given the large gap between thinking and non-thinking configurations in Table 7, could the authors report average per-action inference latency, effective closed-loop control frequency, average reasoning-token count, and total task completion time? How do these figures compare with the VLA baselines on identical hardware?
  • Fairness of the VLA comparisons. Do the VLA baselines receive the same RGB-D observations and geometric information as EASG? Could the authors specify which sensory inputs, action representations, and inference configurations are shared across methods? A controlled comparison under matched information and computational budgets would help isolate the contribution of the grounding interface itself.
  • Evaluation beyond LIBERO. LIBERO is known to inflate scores through task memorization, so results on it may not reflect true generalization. We encourage the authors to additionally evaluate on benchmarks that better probe generalization and zero-shot capability — e.g., RoboLab, LIBERO-Plus, LIBERO-PRO, and RoboDojo — and to report how the gap between EASG, fine-tuned VLA baselines, and the bare VLM without action-space grounding changes in those settings.
  • Interpretation of the zero-success ablations. Table 8 shows total failure when either action-frame grounding or metric-depth grounding is removed. Could the authors offer a finer-grained failure analysis — for instance, whether these variants fail due to wrong movement directions, imprecise distances, or accumulated control error?
Reviewer_pZJ9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Explicit Action-Space Grounding (EASG), enabling pretrained VLMs to directly generate executable robot actions without robot-action pretraining. The method grounds a TCP-centered coordinate frame in visual observations, provides metric depth information, and represents continuous actions through the native textual generation interface. The same framework also supports fine-tuning with robot demonstrations. Experiments demonstrate promising zero-shot manipulation, generalization, cross-embodiment transfer, and competitive fine-tuning results on LIBERO.

Critical strengths
  1. Interesting and well-motivated idea. The paper explores whether pretrained VLMs can directly serve as manipulation policies through an appropriate action interface. I find this direction meaningful, and the proposed framework is simple and effective.

  2. Strong empirical validation. The zero-shot real-world results are particularly impressive, achieving 79.4% success without robot-action training. The experiments further demonstrate robustness under randomization, competitive fine-tuning performance, and cross-embodiment transfer.

Critical weaknesses
  1. Missing inference efficiency analysis. Table 7 shows that performance drops dramatically without thinking (79.4% to 11.7%). Since the method relies heavily on reasoning, inference latency, generated token counts, and execution time are important for evaluating its practicality compared with existing VLA policies.

  2. Limited simulation benchmark coverage. The simulation experiments are conducted only on LIBERO, which is relatively simple and increasingly saturated. I strongly encourage the authors to evaluate on more challenging benchmarks, such as RoboTwin 2.0, LIBERO-Plus, or RoboCasa, to better demonstrate the effectiveness and generalization of EASG. Comparisons with more recent VLA methods, such as StarVLA, VLAct, and LingBot-VLA, would further strengthen the evaluation.

Questions for authors
  1. Could the authors report average inference latency, generated token counts, and total task execution time, particularly compared with conventional VLA policies?

  2. Why does removing either depth or frame grounding lead to complete failure across all tasks in Table 8? Could the authors provide additional analysis or comparisons with alternative grounding representations?

  3. Could the authors consider comparing EASG with more recent VLA methods, such as StarVLA, VLAct, and LingBot-VLA, under comparable evaluation settings and add more simulation benchmarks? (LIBERO is easy)

I am happy to raise my score, if authors loop in the rebuttal actively.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer pZJ9 (Q3)

W2 & Q3. Additional simulation benchmark and comparisons with recent VLA baselines

We appreciate the reviewer's valuable suggestion to strengthen the evaluation with more challenging benchmarks and recent VLA baselines. To further evaluate generalization beyond standard LIBERO, we additionally evaluate EASG on LIBERO-Plus, which introduces systematic perturbations across camera, robot, language, lighting, background, noise, and object layout. We also extend the comparison to more recent VLA models, including StarVLA and VLAct.

Table R5. Generalization performance on LIBERO-Plus under different perturbations. We report task success rates (%). Original denotes the unperturbed setting, and Total denotes the overall success rate across all perturbed evaluation tasks following the LIBERO-Plus protocol.

ModelOriginalCameraRobotLanguageLightBackgroundNoiseLayoutTotal
OpenVLA76.50.83.523.08.134.815.228.515.6
OpenVLA-OFT97.156.431.979.588.793.375.874.269.6
π₀94.213.86.058.885.081.479.068.853.6
NORA87.92.237.065.145.758.612.862.139.0
WorldVLA79.10.127.941.643.717.110.938.025.0
UniVLA95.21.846.269.669.081.021.231.942.9
π₀-FAST85.565.121.661.073.273.274.468.861.6
StarVLA (Qwen3-VL-PI)95.764.357.282.894.294.079.678.277.2
VLAct96.773.968.481.596.796.786.083.382.6
Ours (Qwen3.5-0.8B)83.326.320.157.571.960.372.263.151.9

Despite using a lightweight 0.8B VLM without any robot-action pretraining, EASG achieves an overall perturbed success rate of 51.9%, outperforming several larger action-pretrained VLA models—including OpenVLA (15.6%), WorldVLA (25.0%), NORA (39.0%) and UniVLA (42.9%)—and closely approaching π₀ (53.6%). Notably, EASG remains more resilient to Noise (72.2%) and Camera shifts (26.3%) than NORA, WorldVLA, and UniVLA (which drop to 0.1%–2.2% on Camera), and even surpasses π₀ under Camera (26.3% vs. 13.8%) and Robot (20.1% vs. 6.0%) perturbations.

At the same time, a clear performance gap remains compared with stronger recent VLAs such as StarVLA (77.2%) and VLAct (82.6%), particularly under uncalibrated camera shifts and robot morphology perturbations. Overall, these results show that EASG enables a compact pretrained VLM to achieve competitive robustness without large-scale robot-action pretraining, while clarifying where specialized VLA pretraining still holds an advantage.

Thanks for the authors' detailed rebuttal, i just raised my score.

Reviewer_shPqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Explicit Action-Space Grounding (EASG), enabling pretrained VLMs to directly generate executable robot actions without action pretraining or additional action experts. By explicitly grounding TCP-centered coordinates and metric depth information, EASG bridges visual-spatial reasoning and continuous robot control while supporting further fine-tuning. Experiments demonstrate 79.4% zero-shot success on real-world manipulation tasks and 83.3% success on LIBERO after fine-tuning, with strong generalization and cross-embodiment transfer capabilities. The results suggest that pretrained VLMs can serve as native robotic action models through an appropriately designed action interface.

Critical strengths
  • The idea is straightforward, representing robot actions as structured sequences of numerical values.
  • The paper demonstrates that pretrained VLMs can successfully perform simple robotic manipulation tasks without any fine-tuning.
Critical weaknesses

The proposed EASG method essentially introduces a TCP-based action representation, which has certain limitations, such as its difficulty in extending to dexterous hands. Moreover, several claimed advantages are not sufficiently substantiated, with important baseline comparisons and implementation details missing. Nevertheless, the paper is generally well-structured and presents a relatively complete study. Please refer to the Questions for specific concerns.

Questions for authors
  • In Table 13, the proposed method's average success rate of 67.2% is highlighted in bold, although π0-FAST achieves a higher success rate of 70.0%. Could the authors clarify the highlighting convention or correct this inconsistency?
  • Could the authors clarify the decoding strategy used during inference, including whether greedy decoding or sampling is employed, and specify relevant parameters such as temperature?
  • The paper shows that memory significantly improves zero-shot performance, but provides limited implementation details. What information is stored in memory, how is interaction history maintained, and how is the memory length managed?
  • Could the authors report the inference latency and actual control frequency, particularly for the 27B VLM with thinking enabled? How does its inference efficiency compare with existing VLA models?
  • Why is π0.5 excluded from the fine-tuning and cross-embodiment experiments, despite outperforming π0-FAST in Table 2?
  • Both robots use the same two-finger gripper and matched initial end-effector poses. Are the wrist-camera configurations and viewpoints also consistent across embodiments? How robust is EASG to changes in camera placement and calibration?
  • The paper demonstrates cross-embodiment transfer from UR3 to Flexiv, but does not compare against existing VLA baselines. Could the authors provide comparisons with models such as π0.5 using end-effector pose control to better demonstrate the advantages of EASG?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

I. Acknowledgments

We sincerely thank all three reviewers (shPq, pZJ9, and aF7g) for their thoughtful feedback and constructive suggestions. We especially appreciate their recognition of EASG's central idea: enabling pretrained VLMs to directly generate executable robot actions without robot-action pretraining, while supporting further performance improvements through fine-tuning within the same unified action interface.

The reviewers' feedback has motivated substantial additional experiments and clarifications on inference efficiency, grounding mechanisms, generalization, and cross-embodiment transfer, as summarized below.


II. Key Strengths

We appreciate the reviewers' recognition of the following strengths:

  • Novelty and Motivation: A simple, meaningful approach to enabling pretrained VLMs to serve directly as manipulation policies through explicit action-space grounding (shPq, pZJ9).
  • Zero-Shot Manipulation and Generalization: 79.4% success across six real-world tasks without robot-action training, with strong robustness to object and pose variations (shPq, pZJ9, aF7g).
  • Unified and Transferable Interface: The same textual action interface supports zero-shot manipulation, fine-tuning (67.2% on harder real-world tasks; 83.3% on LIBERO), and UR3-to-Flexiv transfer (pZJ9, aF7g).
  • Clarity and Completeness: All three reviewers positively assessed the presentation, with shPq noting that the paper is well-structured and relatively complete.

III. Key Concerns and Our Responses

Key ConcernsReviewersOur Response
Inference efficiency and runtime.shPq, pZJ9, aF7gAdded runtime comparisons on the same UR3 platform (Tables R1–R2). EASG averages 7.7 steps/task, 583 tokens/step, and 11.6s inference + 3.6s execution per step. Average task completion time is 117.2s versus 39.0–59.2s for VLAs. We explicitly acknowledge this efficiency limitation.
Limited simulation benchmarks and recent baselines.pZJ9, aF7gAdded LIBERO-Plus evaluations across seven perturbation dimensions against nine VLAs (Table R5). Without robot-action pretraining, our 0.8B VLM achieves 51.9% overall perturbed success, outperforming OpenVLA (15.6%), WorldVLA (25.0%), NORA (39.0%), and UniVLA (42.9%), and approaching π₀ (53.6%).
Insufficient grounding failure analysis.pZJ9, aF7gAnalyzed 240 real-world trials across four grounding configurations (Table R4). Robot-Base grounding frequently fails to produce target-directed motion (52/60), while Wrist-Camera and TCP without metric depth preserve target-directed motion but often fail during precise manipulation (39/60 and 46/60). Full EASG achieves 47/60 successes, supporting the complementary roles of frame and depth grounding.
Missing π₀.₅ fine-tuning and cross-embodiment comparisons.shPqAdded π₀.₅ under the UR3-to-Flexiv transfer protocol (Table R3). Although π₀.₅ performs better on UR3 (79.4% vs. 63.9%), its success rate drops 31.1 pp on Flexiv, compared with only 1.7 pp for EASG, demonstrating stronger transfer stability in the evaluated setting.
Fairness of VLA comparisons.aF7gClarified sensory inputs, action representations, and inference configurations. Real-world zero-shot EASG uses metric depth and thinking, without robot-action pretraining. Fine-tuned EASG uses RGB with TCP-frame grounding, without metric depth or thinking.
Missing implementation details.shPqClarified grammar-constrained greedy decoding (do_sample=False), full episode-level interaction memory, and matched wrist-camera configurations with separately calibrated extrinsics. We acknowledge that robustness to camera-placement and calibration errors remains unverified, and corrected the Table 13 formatting error.

We hope these additional experiments and clarifications have addressed the reviewers' major concerns. We respectfully invite the reviewers and Area Chair to reconsider the paper in light of the new evidence. We sincerely thank everyone for their time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.