acceptodds
Withdrawn from ICLR 2027

RoboHarness: A Simple Harness outperforms VLA and World Actions Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

The rise of large language models (LLMs) has opened a path toward generalist embodied AI, raising the question of how LLMs can act in the physical world. One straightforward approach is training the model to generate actions: collect robot-action data and video demonstration, refine LLM architecture with an action head, and train a vision-language-action (VLA) model at scale.However, we argue that coding agents have the potential to realize generalist embodied manipulation. All they need is a simple yet effective robot harness. We present : a robotic harness that gives LLM agents a visual-geometric control panel so that they can directly understand and invoke embodied tasks. With RoboHarness, Qwen3.8 Flash outperforms state-of-the-art VLA models on BEHAVIOR Challenge 2025 tasks. The implementation will be released as open source.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM agent, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pD5DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboHarness connects a pretrained language-model agent to robotic manipulation through an explicit visual-geometric tool interface. The agent selects image keypoints, which are tracked over time using optical flow and depth back-projection. It specifies geometric relations between gripper, object, and target points; the harness searches for satisfying poses, checks inverse-kinematics feasibility, and plans and executes motions. Direct adjustment tools provide additional control over end-effector and chassis motion. The paper argues that the constraint representation forms a minimal and complete interface without additional model training. Evaluation covers the first ten BEHAVIOR Challenge 2025 tasks and a 100-object lifting test, reporting advantages over several VLA and world-action-model baselines.

Critical strengths
  1. A concrete and interpretable interface between visual reasoning and robot control.

Persistent keypoint tracking gives the agent spatial references that can be reused across successive tool calls, while geometric constraints allow it to specify orientation-sensitive manipulation without directly generating every motor command (§3). Separating these specifications from inverse kinematics and motion planning is a practical design choice. The combination of constraint tools and direct adjustments provides a usable route for applying pretrained models without additional action-model training. This is a useful engineering contribution, even though it should not be equated with a new manipulation paradigm or a guarantee of task feasibility.

Critical weaknesses
  1. Limited conceptual novelty.

The central approach largely integrates existing ideas: visual keypoint grounding, geometric constraints, tracking, and conventional motion planning. As the paper itself acknowledges in §2.3, ReKep already generates relational constraints over 3D keypoints and optimizes robot motions, while CoPa uses part-level spatial constraints for grasping and subsequent manipulation. RoboHarness offers a useful interface variant, but this is a limited conceptual advance rather than the claimed new paradigm.

  1. The reported superiority is confounded by human task knowledge and different execution settings.

Appendix A states that the BEHAVIOR prompts are human-written and include task-specific procedures, while Appendix B lists higher-level skills such as opening doors and drawers, picking up objects, and placing objects in containers. The evaluated system therefore includes substantial task guidance and skill scaffolding beyond the advertised minimal primitives. Its results cannot be attributed to the geometric interface alone.

Moreover, the BEHAVIOR comparison places five-instance-per-task results alongside official leaderboard scores without establishing matched evaluation instances. In the 100-object test, RoboHarness uses an R1 Pro with floor objects, whereas learned policies are evaluated in LIBERO tabletop settings (§5.2; Appendix C). Using the same lifting criterion does not equate the manipulation difficulty across these conditions. These experiments demonstrate the behavior of particular complete systems, but do not establish the broad claim that a simple harness outperforms VLA and world-action models because of the proposed interface.

Questions for authors
  1. Beyond the agent-facing API and persistent tracking, what is the principal conceptual advance over the constraint-based manipulation approaches already discussed in §2.3? Which result demonstrates a benefit specific to that advance?
  2. Which reported comparisons use the same embodiment, scene, evaluation instances, and task information? Please clarify which Appendix B skills are used in the evaluated rollouts and whether any existing results separate the contribution of task-specific procedures and skills from that of the geometric tools.
Reviewer_XVPgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RoboHarness connects a general-purpose multimodal model to robot control through image keypoints, persistent geometric state, and constraint-based actions, without additional robot-policy training. The paper evaluates the approach on ten BEHAVIOR-1K tasks with five instances each, adds a 100-object lifting test, and attempts to prove the completeness of its keypoint-constraint representation.

Critical strengths

The paper demonstrates that an LLM can achieve robot control through a unified geometric interface, offering a useful perspective on translating a model's spatial intentions into executable actions and exploring a promising new paradigm for embodied AI. In particular, building this manipulation interface around elementary, composable primitive action tools—instantiated via visual point selection, geometric feedback, and constraint execution—provides a worthwhile design for enabling models to organize complex behaviors. The household-task and object-level evaluations further demonstrate the encouraging preliminary potential of this approach.

Critical weaknesses
  1. The experiments are insufficient to determine whether the core design is effective, outperforms alternative interface designs, or generalizes across broader tasks and environments to support the superiority claims over VLA/WAM. • Limited scope: Evaluation only covers ten BEHAVIOR-1K tasks (five instances each, without repeated trials) and a 5 cm lifting test, lacking multi-stage validation on additional mainstream benchmarks, such as LIBERO-Pro, Robosuite, and RoboDojo. • Lack of core ablations: Primitive action tools are not compared against alternative interface designs under identical settings, nor are keypoint constraints, geometric execution, persistent state, and human procedures/skills isolated. • No physical-robot validation: Robustness to real-world occlusion, perception noise, and contact uncertainty is untested. • Limited foundation models: Only Qwen3.8-27B/Flash and gpt-6-luna are tested; stronger models (e.g., GPT-6Astra, Opus5.5) could be evaluated as supplementary analysis.

  2. The theoretical proofs in Section 4 rely on unstated assumptions and contain specific gaps. Section 4.1 omits sufficient conditions for unique ray intersections on non-convex surfaces, and Section 4.2 overlooks directional ambiguity when moving from line coincidence to directed-vector relations.

  3. The presentation and organization remain preliminary. The manuscript reads somewhat like a technical report. The method section focuses primarily on tool definitions and execution procedures without sufficiently establishing their design rationale, technical novelty, or operating boundaries, and the theoretical analysis and experiments are not yet integrated into a cohesive argument.

Questions for authors

Please refer to the Critical Weaknesses above.

Reviewer_KFMhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RoboHarness, a training-free framework that enables LLM agents to perform robotic manipulation through persistent keypoint tracking, geometric constraints, and low-level control tools. Instead of directly predicting robot actions, the LLM specifies spatial relationships, while the harness handles constraint solving, inverse kinematics, and motion planning. Experiments on BEHAVIOR Challenge tasks and 100-object grasping report promising results compared with several VLA and world-action models.

Critical strengths
  1. The paper explores an interesting alternative to conventional VLA training by combining general-purpose LLM reasoning with geometric robot-control tools. This is a meaningful direction for generalist manipulation.

  2. The reported results on BEHAVIOR tasks and diverse object grasping are promising, suggesting that a well-designed agentic interface can be effective without additional robot-action training.

Critical weaknesses
  1. Unclear contribution of the harness. The proposed keypoint-based geometric constraints are closely related to ReKep and CoPa, while the agentic tool-calling formulation resembles Code as Policies and CaP-X. Moreover, the system relies on human-written task-specific procedures and predefined high-level skills. Without appropriate ablations or comparisons with existing geometric-control agents, it is unclear how much improvement comes from the proposed harness itself.

  2. Insufficient and potentially unfair experimental comparisons. The BEHAVIOR evaluation covers only ten tasks with five instances each. The 100-object experiments compare models operating in different environments and embodiments, making the conclusions difficult to interpret. I suggest including stronger recent VLA baselines, such as StarVLA and VLAct, alongside broader evaluations of world-action models (e.g., OpenWAM-α, and LingBot-VA) under matched settings. Additionally, the paper lacks inference-efficiency analysis, including LLM calls, latency, and computational costs. These are important when comparing agentic approaches with end-to-end VLAs.

Questions for authors
  1. Could the authors provide comparisons with existing geometric-control agents such as ReKep or CaP-X, as well as stronger recent VLA/WAM baselines (e.g., StarVLA, VLAct, OpenWAM-α, and LingBot-VA) on compatible benchmarks under matched evaluation settings?

  2. How much does performance depend on the manually designed high-level skills and task-specific procedural prompts? An ablation using only the primitive geometric tools would help clarify the actual contribution.

  3. Could the authors evaluate more LLM backbones on the BEHAVIOR tasks? The current main results primarily rely on Qwen3.8-Flash-Next, making it difficult to assess generalization across LLMs.

  4. What is the end-to-end inference efficiency, including wall-clock time, number of LLM calls, token consumption, and tool execution overhead? How does this compare with VLA/WAM inference?

  5. Could the authors clarify how action candidates are generated and how the geometric constraint solver and motion planner are implemented?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.