acceptodds
Under review as a conference paper at ICLR 2027

Physical Agency: What the Interface Unlocks

Abstract

Frontier models increasingly act as robot-use agents, calling a robot's motor skills as tools much as they interact with a browser or code editor. But exposing skills as tools leaves open how the model should use them: what it emits, what feedback it receives, and when it decides again. We study the interface through controlled comparisons with the robot, cameras, scenes, and motor-policy weights held fixed, and separately measure the benefit of adding a complementary frozen skill. We introduce Pigey (Physical Agency), an agent that (1) issues language-level actions one at a time over a persistent task state, (2) uses verified execution, checking post-action images and robot state and recovering when it detects failure, and (3) draws on an open set of skills exposed as tools it selects for each subgoal. No additional policy training or demonstrations are required. On 30 real-robot tasks designed to probe decisions over existing motor skills, the same frozen π0.5 policy succeeds in 16.7% of trials when called directly and 84.0% through Pigey. Adding a frozen motion planner as a second skill raises success to 97.3%. Removing verification and recovery lowers success by 28 percentage points, the largest ablation drop. Gains also hold in simulation and across Claude, Gemini, and Qwen reasoners. Most of what the policy was missing was an interface, not a skill.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied AI, reasoning models, physical agents, LLM agents, agent interfaces, multi-modal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tcCBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Pigey, an agent–robot interface for using frozen motor skills more effectively. Pigey executes one language-level action at a time, verifies outcomes using visual and robot-state feedback, recovers from failures, and can select between multiple frozen skills without retraining. The paper validates the design on 30 real-robot tasks and LIBERO-PRO, showing large gains over directly using the frozen VLA, with additional ablations on verification/recovery, skill composition, action granularity, and different VLM reasoners.

Critical strengths

The paper carefully holds the robot, motor policies, and core execution setup fixed while varying the interaction interface, which makes the ablations relatively interpretable.

Critical weaknesses
  1. Limited conceptual novelty. The overall idea of using a high-level VLM/LLM to orchestrate frozen robot skills with feedback and recovery has been explored extensively in prior work, so the core concept is not particularly new.

  2. Limited real-robot task coverage. The real-robot evaluation contains only 30 tasks on a single tabletop setup, which is relatively narrow for supporting broad claims about general physical agency.

  3. The main empirical conclusion is unsurprising. Showing that closed-loop execution with verification and recovery outperforms open-loop execution is largely expected in robotics, limiting the novelty of the main takeaway.

  4. The “interface” contribution is not cleanly isolated. Pigey adds not only a different interface, but also strong VLM reasoning, additional perception, structured feedback, recovery mechanisms, and routing guidance. Therefore, the large performance gain cannot be attributed to the interface alone.

  5. The “open set of skills” claim is weakly validated. The real-robot experiments only compose two complementary skills, whose intended use cases are explicitly described in the prompt. It is unclear whether the approach scales to a larger and more heterogeneous skill library.

  6. The system relies heavily on hand-crafted routing priors. The prompt explicitly specifies when to use TAMP versus VLA, when to retry, escalate, re-perceive, and stop, so part of the performance may come from manually encoded skill knowledge rather than general agentic reasoning.

Questions for authors

See Weaknesses

Reviewer_dESBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how an agent–robot interface enables a vision-language reasoner to use frozen motor skills more effectively. The proposed system, Pigey, selects one language-level action per turn, maintains interaction history, and checks post-action images and robot state to guide verification and recovery. It can invoke either a frozen π0.5 policy or a complementary task-and-motion-planning skill.

On 30 real-robot tasks with five trials each, the raw π0.5 policy achieves 16.7% success, compared with 84.0% when used through the VLA-only Pigey interface. Adding the planning skill increases success to 97.3%. The evaluation includes retry baselines, a closed-loop planner, text-only feedback, single-skill variants, a skill-selection oracle, and a verification/recovery ablation. Simulation experiments compare action-by-action interaction with program generation and evaluate multiple reasoners. On the complete LIBERO-PRO grid, Pigey achieves 72.1% success compared with 53.0% for the frozen policy.

Critical strengths
  1. The real-robot experiments meaningfully separate better use of an existing skill from access to additional skills. The VLA-only Pigey result is particularly important: success increases from 16.7% to 84.0% without introducing the complementary TAMP skill. Similarly, the planner-only agent reaches 83.3%, compared with 48.7% for the raw planner. These comparisons prevent the full system’s 97.3% result from being explained simply by adding a stronger motor backend. The retry baselines and closed-loop planner provide additional context, and the reported task-clustered confidence intervals support substantial differences. For example, VLA-only Pigey exceeds the VLA retry baseline by 57.3 percentage points, with a reported 95% interval of [40.7, 74.0]. This is persuasive evidence that additional attempts alone do not account for the improvement.

  2. The study combines substantial hardware evidence with complementary simulation controls.

The hardware evaluation includes 30 tasks, per-task results for multiple baselines and ablations, and explicit tests of recovery and memory. The simulation program-generation comparison retains the reasoner, tools, perception, policy, and action budget, providing a useful test of when control should return to the reasoner. Improvements appear with both Opus and Qwen configurations. The authors also report the complete LIBERO-PRO grid rather than only the difficult perturbation subset, and acknowledge that gains are small on cells where the frozen policy already performs well. This qualification helps define where orchestration provides the greatest benefit.

Critical weaknesses
  1. The distinct contribution beyond established hierarchical planning and feedback-driven skill execution needs sharper identification. Pigey combines task decomposition, persistent history, outcome checking, and recovery over pretrained skills. These components are closely related to approaches discussed in the paper’s own related-work section. The experiments convincingly show that this combination helps a raw policy, but less decisively establish which aspects distinguish the proposed interface from a well-configured existing hierarchical agent. In particular, the real-robot suite deliberately emphasizes reasoning and memory beyond the short, concrete command distribution of the motor policy. This is a sensible capability-probe design, but it makes the raw-policy comparison primarily evidence for adding a high-level reasoner and suitable subgoals. The text-only planner and program-generation comparisons help, yet the paper should more precisely explain what new empirical conclusion emerges beyond that established architectural principle. The statement that “most of what the policy was missing was an interface” should remain explicitly scoped to these tasks and frozen skills.

  2. Some simulation comparisons do not isolate interface effects as cleanly as the main narrative suggests. Appendix H.1 states that the simulation agent has deterministic grasp/place tools in addition to the frozen VLA. Consequently, the improvement over the raw π0.5 baseline should distinguish additional motor functionality from improved orchestration, as the real-robot single-skill experiments already do. The matched program-generation comparison is stronger evidence for the decision-granularity claim. The direct-control comparison also changes more than action abstraction. Appendix K specifies fixed top-down orientation and a 2 cm controller tolerance, and states that detected objects are provided with coordinates but without labels. This needs reconciliation with the description of matched perception. Conversely, rejecting an incorrect DONE provides the direct controller with an oracle signal. These opposing asymmetries make the 32.0% versus 55.0% comparison difficult to interpret as a clean estimate of the value of skill-level actions alone.

Questions for authors
  1. What exactly is shared between the simulated raw-policy baseline and the agent configurations?
  2. Could you reconcile the direct-control comparison’s observation and controller differences?
  3. How are the feedback and recovery baselines operationalized?
  4. Could you clarify the hardware evaluation and temporal scoring protocol?
Reviewer_C1iSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Pigey, a training-free framework that uses a vision-language model to invoke frozen robot skills, inspect execution outcomes, and recover from failures. Experiments on real-robot tasks and LIBERO-PRO demonstrate improvements over direct policy execution and several alternative configurations. While the system is effective, its methodological novelty and additional empirical insights over existing closed-loop robot agents are insufficiently established.

Critical strengths

1.The paper evaluates the system on physical robot tasks, including disturbances and recovery scenarios, providing practical evidence beyond simulation. 2. Results across multiple model families suggest that the approach is not dependent on a single proprietary model.

Critical weaknesses
  1. The core architecture follows a conventional closed-loop agent design: a VLM calls skills sequentially, retains interaction history, checks execution feedback, and retries or replans. These are established components of robot-agent systems. The paper does not identify a clearly novel mechanism for planning, verification, memory, or skill selection that distinguishes Pigey from existing approaches.
  2. The core architecture follows a conventional closed-loop agent design: a VLM calls skills sequentially, retains interaction history, checks execution feedback, and retries or replans. These are established components of robot-agent systems. The paper does not identify a clearly novel mechanism for planning, verification, memory, or skill selection that distinguishes Pigey from existing approaches.
Questions for authors
  1. What specific mechanism distinguishes Pigey from a standard closed-loop VLM agent equipped with the same skills and observations?
  2. If the intended contribution is primarily empirical, which finding provides new knowledge beyond established benefits of feedback, hierarchical decomposition, and skill composition?
  3. Under matched models, skills, observations, and inference budgets, what evidence shows that Pigey’s particular interface choices outperform a comparably equipped existing agent framework?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.