acceptodds
Under review as a conference paper at ICLR 2027

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

17 views

Abstract

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often degrade under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without fine-tuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by and percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches on RoboTwin C2R. Real-world demonstrations on dual-Franka robots further show target redirection, grasp recovery, and new task compositions with the same frozen VLA. Code is available at https://anonymous.4open.science/r/HarnessVLA2027/README.md.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action, Memory-Guided Agents, Agentic Harness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4YGuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Harness VLA combines a frozen VLA contact primitive with a fixed set of analytic manipulation primitives. A memory-augmented LLM planner uses task-specific traces and global success/failure rules to re-ground tasks, stage the robot, invoke the VLA for contact-rich actions, and recover from failures. The paper reports strong results on LIBERO-Pro, RoboCasa365, and RoboTwin C2R, plus real-world dual-Franka demonstrations and an offline Flash Mode based on compiled task cards.

Critical strengths
  • The division of labor is intuitive: analytic primitives handle grounding, transport, navigation, and release, while the frozen VLA handles local contact-rich actions. This targets a real limitation of both monolithic VLA policies and purely analytic controllers.

  • The trace-level analyses and primitive-usage statistics make the system more interpretable than a black-box success-rate claim. Real-world examples illustrate target redirection, ordered placement, and grasp recovery without fine-tuning the VLA.

Critical weaknesses
  • The novelty over existing code-as-policy and agentic manipulation systems is not fully isolated. The central ingredients appear to be a frozen VLA exposed as a tool, a set of analytic primitives, and retrieval of successful traces plus global rules. The paper should clarify what is algorithmically new and how the memory is constructed, updated, and generalized. Also establish that the harness gains persist across different VLA backbones.

  • Task-specific memory is bootstrapped from a reference seed for each task. This is a reasonable few-shot setting, but it means the strongest benchmark results do not establish transfer to entirely new task identities without preparation. The amount of human intervention or compute needed to create and curate memory is unclear.

Questions for authors
  1. What is the per-task human/compute/API cost of bootstrapping Task Specific Memory and Global Memory, and which parts are generated automatically versus manually curated?
  2. Can the authors provide ablations for memory and planner reasoning, and test the harness with more than one frozen VLA backbone?
  3. What are the API cost for the runs?
  4. Can the real-world demonstrations be quantified with repeated trials and a direct frozen-VLA baseline under the same task conditions?
Reviewer_HK3vOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper describes Harness VLA, a memory-augmented robotic agent that uses a frozen VLA in combination with a predefined set of analytic control primitives. A high-level planner is used to handle semantic grounding, staging, transport, and recovery. However, when it comes to contact-rich interaction, the planner will invoke the VLA. Successful execution traces are stored as task-specific memory and reusable heuristics as global memory to help guide future executions. Additionally, it has a "Flash Mode" that allows for the execution of prepared task cards without having to plan online. Validation was conducted using LIBERO, LIBERO-Pro, RoboCasa365, and RoboTwin; all showed improvements over directly executing the frozen VLA. Most evaluations were performed by constructing memory through exploration on a single reference instance of each task and then testing on other instances of the same task. Some additional evidence was provided through limited zero-shot evaluations, analysis of primitive usage, comparison of different planners, and qualitative demonstrations on dual-Franka robots.

Critical strengths
  1. The paper has been well-written and provides a clear explanation of the framework and each of its parts.
  2. Real-robot demonstrations show how to compose tasks, redirect targets, and recover from failed attempts at grasping.
  3. Evaluation was conducted across several different environments and embodiments (tabletop, mobile kitchen, and bimanual).
Critical weaknesses
  1. The novelty of this approach relative to VoLoAgent [1], which uses an agent to monitor the execution of a VLA and recover from failures when combining it with other tools, is unclear. This related work is neither discussed nor compared to.
  2. The need for a VLA is not demonstrated. There are no comparisons to either a VLM agent acting as a policy or to a strong planner using capable non-VLA manipulation tools.
  3. There are insufficient memory ablations. While task-specific and global memory are key contributions, their individual benefits have not been isolated.
  4. The paper has limited evidence for long-horizon generalization.

[1] VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation, CoRL 2026.

Questions for authors

Please address the points listed in the weakness section. Thank you.

Reviewer_Pz2FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Harness VLA, a framework that exposes a pretrained, frozen VLA as a callable primitive for local contact-rich manipulation. A multimodal planner combines this primitive with a fixed set of analytic controllers for staging, transport, navigation, and release. Task-specific procedural memory is constructed through exploration on a reference instance, while global memory records reusable operating rules and failure knowledge. The framework also supports a Flash Mode that executes compiled procedures with limited predefined recovery. Validation covers standard and perturbed LIBERO, RoboCasa365, RoboTwin clean-to-randomized transfer, and qualitative dual-Franka demonstrations. The main contribution is a practical framework for extending frozen policies through memory-guided orchestration and recovery.

Critical strengths
  1. Substantial improvements over the corresponding frozen policies across several manipulation settings. The matched-backbone comparisons are the strongest evidence in the paper. On LIBERO-Pro, the main Claude Code configuration improves success from 50.0% to 82.4%, while Appendix P reports 92.63% with an additional planner. The strongest reported RoboCasa365 result is 59.20%, compared with 30.0% for the corresponding frozen policy; RoboTwin C2R improves from 50.4% to 58.4%. These results support the practical value of improving deployment-time orchestration without updating the low-level policy, and the evaluation across different embodiments and backbones increases the significance of the finding.

  2. A useful control interface for reusing learned manipulation capabilities. Treating VLA execution as an inspectable, interruptible, and retryable local operation is a sensible design. Analytic staging and transport provide explicit ways to address spatial and semantic changes while retaining learned contact behavior. The procedural traces and failure summaries make the resulting execution structure inspectable and potentially reusable. This is a meaningful systems contribution, particularly for applications where retraining the underlying VLA is expensive.

Critical weaknesses
  1. The experiments do not sufficiently isolate the mechanisms that distinguish the proposed framework. The paper attributes its gains to task-specific memory, global memory, analytic staging, and localized VLA retries, but these components are largely evaluated together. The no-memory comparison on LIBERO-Pro Goal is useful: performance changes from 87% to 31% for position swaps and from 87% to 79% for instruction redirection. However, this does not separate the two memory modules or distinguish staging from additional retries and execution time. The invocation-count curves and final-primitive attribution are descriptive analyses rather than controlled evidence of component necessity. A same-planner comparison against simpler hierarchical VLA execution, together with targeted memory and staging ablations, is important for establishing the specific scientific contribution beyond the effectiveness of the complete system.

  2. The adaptation and feedback conditions are not characterized sufficiently to assess the generalization and comparison claims. The bootstrapping phase permits resets and a generous exploration budget, but the actual interaction cost, budget limits, and handling of unsuccessful reference searches are not quantified. One reference seed therefore does not necessarily imply a small amount of adaptation. Appendix K also describes task-specific exploration for the LIBERO-Pro perturbation settings and all RoboCasa splits, including Composite-Unseen; the results should consequently be interpreted as generalization after reference-instance adaptation. In addition, Appendix A reports simulator-derived contact diagnostics in RoboCasa, which are relevant to the claimed verification and recovery benefits. Clear accounting of adaptation cost, Global Memory construction and freezing, and the contribution of this additional feedback is needed to determine how much of the improvement is attributable to the proposed framework under comparable conditions.

Questions for authors

Questions For Authors

  1. Clarification: What are the exact exploration and deployment budgets, including environment steps, resets, planner calls, and elapsed time? How many attempts are typically required to construct task memory, and how are tasks with unsuccessful bootstrapping included in the reported results?

  2. Clarification: Which tasks and episodes contribute to Global Memory, and when is it frozen? Can any reported evaluation rollout modify the memory used by subsequent evaluation rollouts?

  3. Additional validation relevant to the recommendation: Can the authors provide controlled results using the same planner, observations, and physical-step budget that separately evaluate task-specific memory, global memory, and staging/retry behavior? A representative subset would be sufficient if it directly tests the claimed mechanisms.

  4. Additional validation relevant to the recommendation: How much of the RoboCasa improvement remains when simulator contact diagnostics are removed or replaced with checks based on RGB-D and proprioception?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.