acceptodds
Under review as a conference paper at ICLR 2027

Show-Harness: VLM Agents Can Play Robots

32 views

Abstract

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an *Embodied Harness* that enables VLMs to **"play" robots** through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for each step's physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI, a GUI-based manipulation interface that enables humans and agents to "play" robots and collect demonstrations in the same semantic action space across embodiments. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Model, Multimodal Agent, Embodied Harness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dPEROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on that basic vision-language models are hard to translate to robot control though it has world knowledge, and proposes Show-Harness, a model-free semantic action interface. It compresses the control space into 15 discrete semantic units: 6 translation units, 6 incremental rotation units, 1 GRASP/ 1 RELEASE, and 1 DONE. The model only chooses the direction of movement, while the movement range is determined by translators written for each robot, which also enforce workspace and safety constraints.

Then, the paper builds a closed-loop shell consisting of 9 plugins from perception, reasoning, and action, and presents two usage modes: ZS uses Gemini-3.1 Pro to directly output semantic units via prompts with no training, and FT fine-tunes only the linear layers of Qwen 3.5-2B using LoRA (about 3% of parameters, with the visual encoder frozen).

However, there are already works like Guava (2026), which proposes a general Harness framework and semantic action abstraction, and DSD (2026), which decomposes direction/scale at the action token level. These works are not discussed in the Related Work, nor included as compared experiments.

Critical strengths

Main results show average success rates across tasks of ZS 89.0% and FT 86.0% (strongest baseline RATS 57.0%), across environments 100.0% and 88.0%, and across robots 93.0% and 87.0%. The paper also provides horizontal ablations for six types of action interfaces, cross-design experiments between the interface and the shell, factor experiments for action naming and rule prompts, and step attribution experiments. References are timely, and comparison methods are taken from work in the past two years.

Critical weaknesses
  1. Key conclusions lack uncertainty and pairing significance. The main table contains only single-point estimates with 10 trials per task and only 20 trials across environments. At 10 trials, the 95% normal approximation half-width for individual success rates can reach approximately 0.3, while the Wilson interval half-width is about 0.26. The difference between 6/10 and 10/10 for the same model cannot determine whether it stems from task effects or sampling fluctuations, yet the summary employs strong terms like "generalize robustly." It is recommended to supplement uncertainty intervals for each task and include paired tests and effect sizes under shared layouts for key comparisons, thereby narrowing the summary conclusions accordingly..
  2. The action interface ablation lacks external method comparisons. Table 3's interface ablation only compares the L0–L4 abstraction layers, without incorporating complete strong baselines such as π0.5, GR00T, H-VLA, G-VLA, CaP-X, and RATS under the same control variables for interface-level comparison. All six layers were implemented by the authors, with their performance ceiling depending on prompt format, action granularity, and skill library coverage, making it difficult to rule out the possibility of weaker implementations serving as controls. After introducing specialized insertion skills for high-level skills in the paper, the fine-grained task performance improved from 42.5 to 62.5. It is recommended to include at least one published external method with similar interface hierarchy, retraining the controls under the same task and budget conditions.
  3. The metric "step" has inconsistent cross-method definitions. One step for semantic agents corresponds to a single model decision (a semantic unit), while π0.5 and GR00T adopt the "predict 8 action chunks, execute 4 and replan" mechanism (A.2.9). The main text never defines whether their "step" refers to the number of replans or the number of executed actions at 10 Hz. The 50-step limit and timeout failures thus represent different physical execution volumes for the two method types, raising doubts about the comparability of success rate figures. It is recommended to clarify the definition of "step" for each method and unify the limit criteria, or align comparisons under the same model call budget.
  4. The fine-tuning mode underperforms the zero-shot mode in fine-grained and reasoning tasks. After reducing the step size, the zero-shot performance improved from 60% to 80%, while fine-tuning only increased from 40% to 65%; in reasoning tasks, the two modes scored 85% and 10%, respectively, with fine-tuning reaching only 70% even after external subtask instructions. The paper does not explain this discrepancy. Since low-cost deployment is one of the two core claims, it is recommended to supplement a detailed comparison of the two modes under identical prompts and interfaces, as well as their performance on reasoning probes before and after fine-tuning, to distinguish between the backbone capacity limit and the erosion of instruction-following capability.
  5. The version reproducibility of closed-source models requires clarification. The paper itself demonstrates that the directional mapping accuracy of open-source models with 300 billion parameters under the same interface is significantly lower than that of state-of-the-art interface models, indicating that the benefits of the interface still largely depend on the backbone strength. Since such services cannot be frozen long-term, version information is essential for assessing transferability. It is recommended to list the complete version identifiers and access times for each model, along with at least one minimal reproducible configuration based on publicly downloadable backbones.
  6. Expressions such as "cross-robotic" and "capable of operating robots" exceed the scope of evaluation. The actual assessment is confined to desktop single-arm and dual-arm parallel gripper grasping and placing tasks, and the paper acknowledges that it has not yet been validated on humanoid robots or dexterous hands. However, the title and conclusions employ broader coverage, including terms like "expensive robot-specific pre-training." The more diverse scenarios in the appendix are qualitative examples. It is recommended to limit the conclusions in the abstract and introduction to desktop parallel gripper grasping and placing tasks and explicitly enumerate more complex robots as limitations.
  7. Two settings of cross-object migration are presented side by side but without emphasizing the differing strengths of migration evidence between them. The zero-shot mode only requires replacing the translator to switch to another robot, while the fine-tuning mode jointly trains on data from both robots, representing migration evidence of different intensities. Additionally, the paper has observed that the migration loss for orientation-related units is significantly lower than that for grasping-related units, which serves as strong supporting evidence for its design claims yet is only briefly mentioned. It is recommended to move the mechanism of defining orientation in the image coordinate system and its sub-item data to the method section, along with detailing the composition of training data for each row.
  8. The residual failure of this article itself has not been attributed for analysis. The paper breaks down the failure of competitive methods into specific causes, but it still has about 10% failure in cross task analysis and does not provide an equally granular composition; Due to the mechanistic claims of gradually correcting errors and terminating invalid searches through interfaces, the distribution of residual failures directly determines their functional boundaries. Suggest supplementing the proportion of failure modes by task category and distinguishing interface limitations from sensing limitations such as lack of force and tactile feedback.
  9. The default activation strategy and respective costs of plugins have not been fully disclosed. The paper has provided independent contributions by closing each plugin one by one, but has not explained the model call and word element overhead brought by each plugin. This not only affects the selection in resource constrained scenarios, but also makes it difficult for readers to judge the level that can be achieved with the minimum engineering investment. Suggest adding the average number of steps, model calls, and inference latency when each plugin is enabled and disabled.
  10. The human baseline lacks expert remote control. Among the three operators, one without robot experience completed the vast majority of tasks after 15 minutes of explanation and 5 practice sessions, but lacked the reference of skilled operators, and this score may only reflect the lower difficulty of the tasks. Suggest adding a set of experienced operator results using conventional remote operation or graphical end control.
  11. Several technical details need to be clarified. Firstly, the step size is inconsistent between the main text and the appendix, and a complete set of steps and switching criteria need to be provided; Secondly, the paper states that the demonstration can be reused across ontologies, but the real machine corpus is recorded with a uniform step size. It needs to be explained how to maintain consistent displacement when the step size does not match; Thirdly, the paper highlights the relative advantage of the visual language action model, which does not require action heads or special lexical elements. However, similar works that output discrete motion lexical elements are also listed. It is suggested to rephrase this as not requiring new action tokenizers and regression heads.
Questions for authors

None.

3 comments

It added some experiments or only some tables, but "only add", also some new problems were brought from them. Also some issues in the summary is not answered.

Follow-up Response to R3 dPER

We sincerely thank the reviewer for raising the score and for the follow-up feedback. Our initial response focused on reporting the experimental results for each point; below we connect the core insights across these results and clarify how the issues in the Paper Summary are addressed.

Paper Summary and W2: Guava, DSD and Published Systems

The comparisons with Guava and DSD (Sec. 2, Tabs. 3 and 7) share one insight: our gain over both comes from leaving motion magnitudes to the interpreter.

  • Guava. At the framework level, Guava couples its harness with external perception models (SAM3 segmentation and depth back-projection) and object-centric tools mixed with metric poses. In Tab. 3, these lift Guava above our bare-pose layer L1 (65.8% vs. 50.0%), confirming their value for coarse transport (87.5%, against 97.5% for ours). On fine tasks, however, object tools cannot bypass sub-centimeter alignment: Guava must still output metric coordinates and drops to 37.5%, whereas Show-Harness, with no external perception model, reaches 80.0% by keeping the VLM in a closed directional loop and leaving step sizes to the interpreter.
  • DSD. DSD decouples direction and scale at the action-token level. Rebuilt in our training pipeline (Tab. 7), this factorization lifts binned tokens from 8.9% to 30.0%, confirming the benefit of separating direction, while our semantic units reach 66.1% because the model no longer predicts a scale at all.

W4, W5 and G1: How ZS and FT Complement Each Other

Rather than competing pipelines, ZS and FT play complementary roles on the shared action space, as the new probe, open-weight and dynamic-task results show:

  • ZS (Gemini-3.1 Pro) brings frontier reasoning with no robot data: 85% on reasoning tasks and 91.5% on the reasoning probes.
  • FT (Qwen3.5-2B) is a minimal reproducible open-weight configuration that runs locally at 39 ms per decision. It nearly matches Gemini on the Plate tasks (45/50 vs. 48/50, W5) and leads on moving targets (45/60 in simulation and 17/20 on the truck, against 31/60 and 9/20 for π0.5, G1).
  • Fine and reasoning tasks (W4). Adding back the inputs omitted by FT's minimal prompt narrows the gap to ZS on both fine tasks (65% → 70% with proprioception text, vs. 80% for ZS) and reasoning tasks (10% → 70% with the planner's subtasks, vs. 85% for ZS). The probes before and after fine-tuning (2B: 47.0% → 43.0%; 9B: 63.5% → 61.0%) show that the remaining reasoning gap comes mainly from the small backbone's capacity, while LoRA fine-tuning costs only 2.5 to 4 points.

W1, W3, W6 to W11: Plugin Costs and the Revised Manuscript

  • Plugins (W9). Enabling Multi-View Guidance, Proprioception, Action History or Failure Recovery reduces both calls and episode time (28 to 34 → 26 calls; 5.6 to 7.0 → 5.4 min), because early state feedback and grasp rollback prevent oscillation and unguided wandering after missed grasps.
  • Manuscript. The revised manuscript defines steps and timeouts for every method (Sec. 4.1), explains cross-embodiment transfer (Sec. 3.2, App. A.2.10) and reports paired tests and intervals (App. A.3.3). The abstract, introduction and Limitations now state the tested scope (tabletop manipulation on single- and dual-arm parallel-jaw robots), and the abstract's cross-task, cross-environment and cross-embodiment claims rest on these tests (every gain over the strongest baseline is statistically significant, p < 3e-4).

If any specific question remains, we would warmly welcome the chance to address it. The detailed recommendations have already helped us strengthen the paper substantially; we hope the analyses above and the revised manuscript resolve the remaining concerns, and would be grateful if they could be considered in the final assessment.

Final Manuscript Update to R3 dPER

We sincerely thank the reviewer again for raising the score and for the detailed recommendations. Before the discussion closes, we would like to briefly note one final update in the uploaded PDF:

  • Following the recommendation to align the conclusions with the evaluated setting, we have further updated Sec. 4.3 and the Conclusion, which now match the abstract, introduction and Limitations in consistently stating the tested scope (tabletop manipulation on single- and dual-arm parallel-jaw robots).

We will also be happy to incorporate any further suggestions into the final version of the paper. We are grateful for the thorough feedback, which has helped us strengthen the paper, and would be deeply grateful if the updated manuscript could be reflected in the final assessment.

Reviewer_sgfSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents Show-Harness, an "embodied harness" that lets VLMs control robot arms directly through a small discrete action vocabulary: single-axis end-effector translations (MV_FWD/BACK/LEFT/RIGHT/UP/DOWN), incremental rotations, GRASP/RELEASE, and DONE. An embodiment-specific interpreter deterministically maps each unit to a bounded Cartesian setpoint update, so the VLM only chooses the direction and the interpreter sets the magnitude. A set of perception, reasoning, and action plugins (multi-view guidance, proprioception text, subtask planning, adaptive step size, action chunking, action history, failure recovery) closes the perceive–reason–act loop. The same interface supports two modes: (i) zero-shot control by frontier VLMs (Gemini-3.1 Pro by default), and (ii) LoRA fine-tuning of small VLMs (Qwen3.5-2B) on demonstrations collected through GUMI, a keyboard/GUI interface that both humans and agents can operate. On ten real-robot pick-and-place tasks on a Franka and a bimanual AgileX, both modes outperform VLA baselines (π0.5, GR00T), VLA-centric agents, and code-as-policy agents across task, environment, and embodiment shifts (e.g., 89%/86% vs. 39% for π0.5 on cross-task). The appendix adds controlled interface ablations (semantic units vs. numeric deltas, absolute poses, image points, and high-level skills), plugin ablations, cross-embodiment transfer without target data, LIBERO-Plus results (82.6%), and a cost analysis.

Critical strengths
  1. The interface is studied with unusually controlled ablations. The paper goes beyond reporting system-level gains and isolates why the interface works. With the same model, observations, and budgets, semantic units outperform numeric displacement, absolute poses, image points, and scripted skills (Table 3). The advantage persists at every harness level (Table 4). Semantic naming matters in the zero-shot setting but much less after fine-tuning (Tables 5 and 7). Taking the step magnitude away from the model is the dominant factor (Tables 6 and 7), and the authors support this with paired McNemar tests and Wilson intervals. These experiments yield a transferable finding: VLMs choose directions reliably but estimate metric magnitudes poorly. This is the main reason I view the paper favorably.

  2. The real-robot evidence is substantial. All main results come from physical robots on two embodiments, with held-out objects and environment shifts. The appendix adds sim-to-real experiments, transfer to an unseen arm (32/50 vs. 4/50 for π0.5, Table 9), checks on baseline fairness (GUMI data vs. teleoperation data, Tables 7–8), and a full accounting of latency and cost (Table 12). For a robot-learning paper, this weighs strongly in its favor.

Critical weaknesses
  1. The discrete, quasi-static action space limits what the approach can express, and the evaluation stays within that limit. Every action is a single-axis, fixed-step position increment followed by observation, with a binary gripper and no control over velocity, force, or impedance. This excludes dynamic or contact-rich behavior (wiping, pouring, insertion under force, throwing, catching moving objects) as well as dexterous or whole-body control. Precision is bounded by the interpreter's step size, and coordinated multi-axis motion appears only through a plugin that composes units. The zero-shot mode is also slow (8–20 s per step, about 5 minutes per episode), so it can handle only quasi-static scenes. The task suite fits these assumptions closely: it consists mostly of tabletop pick-and-place with cubes, fruit, and a plate or bowl, plus stacking and one tight peg task. The VLA baselines are further handicapped. They are trained on about 9 demonstrations per task of stepwise, interpolated GUMI trajectories with zero-velocity pauses, which is a low-data, piecewise regime that favors the discrete interface. The claim that "the right interface unlocks substantial embodied capability" is therefore demonstrated only where the interface's expressiveness is sufficient by construction. The paper does not test tasks where continuous, reactive control is necessary, and the limitations section mentions only tactile and force sensing.

  2. Robustness comes from hand-engineered per-episode heuristics, not from a mechanism that improves over time. Much of the zero-shot performance depends on manually written plugins and prompt rules. Examples include "NEVER OSCILLATE," fixed descend/lift hints, grasp-affordance rules, and an empty-grasp detector with a scripted retry. Table 4 shows that semantic units alone reach only 43% and that guidance plus plugins add more than 50 points. Exploration, failure diagnosis, policy improvement, and skill accumulation are each either hand-coded or absent. Every episode starts from scratch with the same fixed prompt. Failures are not used to revise rules or the policy. The fine-tuned model is trained once on human/agent demonstrations and never updated from its own rollouts. The one step toward autonomy, self-discovered rules from 12 exploratory probes (Table 5), performs clearly worse than the stated rules. This sits uneasily with the "play" framing and with the scalability claim. GUMI already lets agents collect data in the same action space, so the paper could have closed the loop: autonomous exploration, failure diagnosis, retraining of the small VLM, and accumulation of reusable units or rules. Without this, extending the system to new tasks or failure modes still requires human prompt engineering and new demonstrations.

Questions for authors

How much task- or object-specific prompt engineering was used across the evaluated tasks, for example affordance rules, completion criteria, and step-size thresholds such as the 0.25 cm switch for peg-tight? Were any prompts tuned on the evaluation layouts?

Reviewer_mDi6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Show-Harness, a framework that enables VLMs to control robots through discrete directional, rotation, and gripper commands. Embodiment-specific interpreters translate each command into a bounded motion of fixed step size, while planning, feedback, and recovery plugins support closed-loop control. The framework supports zero-shot frontier VLMs and fine-tuned smaller models, with a shared GUI for control and demonstration collection. Experiments on Franka and AgileX robots and on LIBERO-Plus evaluate task and environment generalization, embodiment transfer, and action-interface design.

Critical strengths
  • Ablations examine semantic naming, prompt guidance, harness support, and step-size prediction. The results suggest benefits from decoupling direction and motion magnitude (66.1% success with semantic units versus 38.9% with free numeric steps), while semantic names appear less important after fine-tuning. Test-time step-size adjustment also supports finer control without retraining, providing useful guidance for VLM–robot interfaces.
  • Experiments cover two robot embodiments, environment shifts, precision tasks, and unseen-arm transfer, alongside LIBERO-Plus evaluation and runtime accounting. Episode-matched teleoperation comparisons help assess whether the demonstration interface disadvantages VLA baselines. Together, these studies support the approach’s practical value.
Critical weaknesses
  • Limited evidence for broader manipulation generalization. Most tasks involve relatively static tabletop manipulation with parallel-jaw grippers. Precision and coordination tests help, but dynamic, time-critical, and more complex contact-rich control remain untested. Multi-minute episodes in zero-shot API mode further limit the demonstrated operating regime.
  • The source of the advantage needs qualification. The interpreter sets motion magnitudes, while VLA baselines predict full metric actions. After fine-tuning, arbitrary labels nearly match semantic names (63.3% vs. 66.1%), suggesting that semantic naming provides a modest additional benefit. Broader claims of semantic preservation and superiority over VLAs therefore remain less established.
Questions for authors
  • In fine-tuned mode, how are step sizes chosen during evaluation, and how much task-specific configuration does this require?
  • Beyond Table 7, is there evidence separating the benefits of discrete direction selection from externally chosen step sizes?
  • Do you have evidence on performance with moving targets or disturbances during API latency? If not, please clarify the scope of the generalization claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.