acceptodds
Withdrawn from ICLR 2027

Robo-Harness K1: Harnessing Robot-Use Agent via Perception Augmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Foundation vision-language models (VLMs) recognize objects, interpret instructions, and reason about spatial relations. A central question for robotic manipulation is how to fully translate this capability into physical actions. Adding a learned action head, as in vision-language-action (VLA) models, requires large-scale demonstrations, may degrade the inherent VL understanding of VLMs, and generalizes poorly. Recent results of Astra (GPT-6) as policy show the potential of robot-use agents (RUAs), which keep the VLM intact as a direct controller. However, this route is slow, expensive, and difficult for weaker models. We step back and ask: what is fundamentally missing between a capable VLM and a working manipulation policy? We find that the answer is accessible perceptual toolkits for 3D spatial understanding and measurement. We introduce Robo-Harness K1, which exposes perception as tools for harness: the agent queries calibrated depth, inspects visual anchors with persistent object identities, evaluates grasp hypotheses, and makes actions and motions from the returned evidence. Robo-Harness K1 enables the VLM to understand depth and space without invasive depth-encoder retraining, so the agent can reason about 3D object relations instead of guessing. On LIBERO-PRO, Gemini 3.7 Flash with K1 reaches 77.8%, surpassing GPT-6 Astra's 61.1% with an RGB-only harness, and K1 further augments Astra to 88.9%. Without target fine-tuning, Gemini with K1 transfers to three RoboSuite arms (90.0% average on shared tasks) and dual-arm RoboTwin tasks. For training RUA models, K1 provides better sample efficiency and generalization than VLAs and vanilla RUA. A Qwen3.5-9B model fine-tuned on only 107 episodes reaches 44.2% on new-state generalization versus 30.2% for OpenVLA, and 13.9% on new-task generalization versus 0.0% for OpenVLA. These results suggest that training RUAs with a proper perceptual harness outperforms VLAs, unlocking VLMs' great potential for robotic manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, robot-use agent, vision-language model, agentic policy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7qvoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides a suite of tools for VLM-based robot control systems, including spatial localization, depth measurement, object tracking, and grasp candidate generation, enabling the model to execute operations based on tool feedback. The trajectories resulting from successful tool invocations are then used to fine-tune smaller Qwen models. The authors evaluate the approach across LIBERO-PRO, RoboSuite, and RoboTwin, covering scenarios such as the deployment of frozen models, student model training, and cross-robot transfer.

Critical strengths
  • The authors distinguish between visible surfaces and the full pose of an object, between measurement hypotheses and executable grasps, and between historical identifiers and currently valid coordinates. These design choices help mitigate the risk of a VLM treating uncertain perceptual outputs as definitive facts that are a feature of practical value for real-world systems.

  • This paper is well-structured, clearly presents detailed information about the model, and provides plenty of clear figures and diagrams. The technical details in the supplementary material effectively complement the main text.

Critical weaknesses
  • Some aspects of the experimental design are not fully considered. In the RGB baseline, the episode terminates immediately upon a done signal, whereas K1 receives a native "task success" determination and allows for corrective actions in the event of failure. The RGB-only Astra baseline uses different control, memory, and completion-feedback mechanisms, while the RGB-only Qwen RUA baseline has fewer observation inputs and suffers from frequent invalid action outputs. These differences make it difficult to isolate the contribution of the proposed perception tools.

  • The transferability to real-world hardware remains unclear. The experiments rely on two somewhat idealized assumptions: first, that the simulation environment pauses during model inference and tool invocation, which does not apply to the real world; and second, that the done signal, indicating task completion, is directly available from the simulator, whereas such feedback is typically absent in real-world settings.

Questions for authors

Q1. Fairness of experimental comparison. The RGB baseline terminates upon done, while K1 receives task-success feedback and can continue execution. Could the authors provide an ablation to isolate the contribution of perception tools from differences in feedback, control, and memory?

Q2. Fairness of the RGB-only RUA baseline. The RGB-only RUA baseline has fewer observation inputs, and some of its evaluations fail due to invalid outputs. Would matching the input modalities and using constrained decoding lead to a fairer comparison?

Q3. Real-world applicability. The experiments pause simulation physics during inference and perception queries and rely on simulator-provided task-completion feedback. How would the system handle inference latency and task-completion detection on real robots?

Reviewer_GRA1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ROBO-HARNESS K1, an agentic framework that leverages VLMs and perception tool calls to perform manipulation tasks. On selected benchmarks, they show that their framework outperforms end-to-end learned behaviour generation policies.

Critical strengths

Although not novel, incorporating perception as tool calls into agentic frameworks is promising.

Critical weaknesses
  • Novelty: Combining existing perception models in an agentic framework is not novel and has been explored before
  • It would have been interesting to see what components actually drive performance. I.e. which perception modules show highest gains and what are prominent failure cases? Without these, this paper provides little to no insights in the actual mechanisms that make the framework work.
  • K1’s done queries the simulator’s success check and allows retries (142 calls), whereas in the RGB baseline done terminates the episode. This is privileged information unavailable at deployment
  • Cross-embodiment evidence is weak: UR5e/IIWA are tested only on four tasks (three saturated at 100%), with the same Panda gripper and Cartesian controller
  • Student can't reproduce half of training episodes (51% SR In Domain), perturbed initial states as in B can not be phrased real OOD and C is on 36 rollouts.
  • No real robot experiments, which would be especially interesting when leveraging large-pretrained models zero-shot and under real-world challenges faced by foundation models (harder perception, noisy inputs and outputs).
  • Confounded and statistically weak evaluation protocol: Only 18 rollouts for Astra-6 can not give statisistcally meaningful results. Also, the evaluation setups are cherry picked (Pos and Task) where VLAs fail. Other perturbation axes have higher VLA success rates and are not compared to K1. Also, number of rollouts differ per method and compared to the baselines. Perception, memory, motion interface, and completion feedback change together in Table 5.
Questions for authors
  • How does performance change when individual tool families are removed, and when done terminates the episode as in the RGB baseline?
  • How does Qwen3.5-9B with the K1 harness perform zero-shot (before fine-tuning), to separate the effect of the harness from that of the SFT?
Reviewer_xysqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Robo-Harness K1, a perception-augmented framework for Robot-Use Agents (RUAs). Its central idea is to expose region grounding, depth measurement, geometric fitting, persistent visual tracking, and grasp proposal generation as callable tools, rather than modifying the VLM architecture or training a separate action head. The VLM uses RGB observations, robot state, and tool-provided 3D evidence to invoke generic motion commands for robotic manipulation.

Critical strengths
  1. Without any target-task fine-tuning, the K1-equipped Gemini successfully achieves zero-shot transfer across environments and embodiments。

  2. The compact Qwen3.5-9B model, fine-tuned using K1 tool-call trajectories, requires only 107 teacher episodes to surpass action-policy baselines like OpenVLA and π0.5.

Critical weaknesses
  1. The experiments and findings are completely restricted to a simulation-only setting. The paper lacks deployment and validation on real physical robots.

  2. The paired evaluation is too small. The key Astra comparison includes only 18 cases, with 11/18 successes for RGB-only Astra and 16/18 for K1+Astra.

  3. The paper does not consider efficiency issues, lacks a comparison of efficiency across different methods.

Questions for authors
  1. The K1 relies on a series of serialized tool calls. In a real, high-frequency robot control loop, how does the latency of these multi-turn API queries compare to a VLA model with a single forward pass?

  2. The experiment has only 18 cases,please provide a larger scale evaluation results.

  3. Please provide the evaluation on on real physical robots.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.