acceptodds
Under review as a conference paper at ICLR 2027

Recursive Harness Distillation across Agents for Robot Manipulation

Abstract

A central goal in robotics is to enable manipulation across changing tasks and environments. Vision-language-action (VLA) models provide broad manipulation capabilities but can struggle when execution requires diagnosing failures and adapting behavior. Strong agents can discover effective interventions through interaction with these policies. We propose Recursive Harness Distillation to accumulate this experience as reusable guidance across agents. A strong agent distills its experience into a playbook for a light agent, then recursively refines the playbook using the light agent’s execution feedback. The resulting playbook enables agents to reuse accumulated intervention knowledge in new task instances without updating model parameters. In real-world manipulation, the harness improves success from 37.3% to 64.0%. On SimplerEnv Bridge, the light agent with the playbook achieves 66.7% success, compared with 41.7% for the GR00T-only baseline, and outperforms the strong agent without a playbook. The same playbook also benefits the strong agent, which reaches 79.2% success. These results demonstrate the feasibility of harness distillation for robotics: intervention experience can be accumulated, refined through execution, and reused across agents to improve manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, vision-language-action models, robot manipulation, agent harnesses

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_czTtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes the Recursive Harness Distillation (RHD) framework to improve the execution reliability of vision-language-action (VLA) models in robotic manipulation tasks. The core idea is to enable a strong agent to distill its execution experience into reusable Playbooks and recursively refine them based on feedback from a lightweight agent. These Playbooks improve VLA performance through instruction, attention, and action interventions without updating model parameters.

Experiments are conducted in SimplerEnv Bridge and on real robots. In simulation, the task success rate of the lightweight agent increases from 43.8% to 66.7% with the optimized Playbooks. On real robots, the success rate improves from 37.3% to 64.0%. Ablation studies further demonstrate the importance of different intervention methods, recursive refinement, and guidance from a stronger model. These results suggest that the proposed framework effectively transfers execution experience across agents.

Critical strengths

S1. The method offers practical value. The paper proposes transferring experience across agents through recursive Playbook refinement without updating model parameters. This provides a low-cost and easy-to-deploy approach to improving the execution capabilities of existing VLA models.

S2. The experimental results are convincing. The method achieves significant improvements in both simulation and real-world robotic tasks, reaching success rates of 66.7% and 64.0%, respectively. These results demonstrate the effectiveness of the framework in simulation and its potential for real-world applications.

S3. The ablation studies are relatively comprehensive. The paper examines the contributions of recursive refinement, teacher model capabilities, and different intervention methods. It also compares deployment costs. These experiments help explain the sources of performance improvements and support the design choice of using a stronger model to guide a lightweight agent.

Critical weaknesses

W1. Insufficient baseline comparisons. Although the paper compares initial and optimized Playbooks, it lacks direct comparisons with methods such as non-recursive experience summarization and traditional prompt optimization. Therefore, it is difficult to determine how much of the performance improvement comes from the recursive distillation mechanism itself.

W2. Limited evaluation of generalization. The experiments cover only four simulation tasks and three real-world robotic tasks. Moreover, the Playbooks require task-specific adaptation in real-world settings. It remains unclear whether the method can effectively generalize to different robots, VLA models, and more complex tasks.

W3. Unclear practical costs and stability. The paper mainly compares inference costs during deployment but does not fully quantify the total cost of strong model exploration and multiple rounds of Playbook refinement. In addition, the initial Playbooks actually reduce the success rate, suggesting that the method may be sensitive to the quality of the distilled experience and the refinement process.

Questions for authors

Q1. Have the authors considered comparisons with other Playbook optimization methods, such as one-time experience summarization or simple prompt optimization? How can they demonstrate that the performance gains mainly come from recursive distillation rather than additional experience accumulation?

Q2. How many iterations does recursive Playbook refinement require to converge? Are there significant differences in convergence speed and final performance across tasks?

Q3. In the real-world robot experiments, the Playbooks require task-specific adaptation by a stronger model. How much additional experience does this adaptation require? How much of the final performance improvement can be attributed to knowledge transferred from the original Playbooks?

Q4. The paper mainly compares inference costs during deployment. Could the authors also report the total costs of strong model exploration, Playbook generation, and multiple refinement rounds to better assess the practical cost-effectiveness of the method?

Q5. The initial Playbook reduces the success rate from 43.8% to 31.3%. Could the authors explain the main reasons for this performance drop and how the recursive refinement process prevents incorrect experiences from causing further performance degradation?

1 comment

Thanks to the authors for the detailed responses and additional experiments. Most of my concerns have been addressed. However, it is still unclear how much of the improvement comes from recursive refinement itself rather than simply using more execution experience. The total development cost is also not fully reported, and the stability of the refinement process remains a concern. Overall, I appreciate the additional results, but I will keep my rating unchanged at Weak Acceptance.

Reviewer_6PGNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose to distill expensive VLMs into a harness for a lighter VLMs for robotic control. To do so, the agent looks at its experience, writes down its learnings and findings in a playbook text file, and the light agent consults this and acts. This is iteratively repeated to update the playbook to maximize the light agent's success. The framework is tested on real world pick and place tasks and in sim.

Critical strengths
  • the idea of distilling a more expensive VLM agent into a more lightweight vlm through writing a harness is interesting and timely.
  • the paper is generally well written.
Critical weaknesses

The idea of having the stronger agent distill its experience into a playbook file for a lighter agent is interesting, but this also somewhat obvious and easy for LLMs / VLMs to automatically do, not quite meeting my bar for intellectual contribution at a venue like ICLR (my opinion, other reviewers / AC may be different).

I think that strong models are quite capable of automatically doing this themselves. Especially if we use models with orchestration abilities that can dispatch lighter weight subagents to reduce costs and latency. We just need to prompt them.

Given this, this paper would be stronger if it attempted to maximize the savings in tokens and latency, compared to a VLM / LLM that was simply prompted to come up with cheaper controllers for a robot task.

Questions for authors

See weaknesses, what sets this paper apart from prompt engineering efforts?

How can you deliver more insight to the research community apart from just setting up essentially a prompt for a VLM to distill a cheaper lighter weight VLM controller? Could you show your specific approach is more efficient, how does the distilled controller vary in token usage / latency, etc.?

Reviewer_vSPUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a harness for operating a frozen VLA through an external LLM. The harness offers three intervention methods: changing the text instruction, biasing the VLA’s visual attention toward an image region, and replacing the output actions. The external LLM decides when and how to intervene, guided by a markdown playbook that a stronger LLM (Astra) writes and recurrently revises based on how a lighter LLM (Luna) performs with it. With the playbook, Luna improves success from 43.8% to 66.7% on four SimplerEnv Bridge tasks, and from 37.3% to 64.0% on three real-world Franka tasks.

Critical strengths

A complete harness with three intervention methods acting at the input, inside the model, and at the output of a frozen VLA, operated by an external LLM.

A practical idea: a cheap LLM reuses a strong LLM’s experience through a recurrently revised playbook, without updating model parameters, at about 1/5.5 of the deployment cost.

Real-robot results and informative negative findings, such as a Luna-written playbook hurting performance (22.9%).

Critical weaknesses

No comparison with other harness methods. The method is compared only with its own ablations. Existing harness methods, such as Code as Policies [ICRA 2023] and ExpeL [AAAI 2024], should be compared under the same LLM agents and budget. Without these comparisons, it is unclear whether the gains come from the proposed harness or simply from having a strong LLM in the loop.

Unclear novelty. The method appears to be a combination of existing techniques: instruction rewriting through text prompts, inference-time attention steering, direct action overrides, and a markdown file of accumulated experience. Each component closely resembles prior work, and the paper does not clearly show what new capability arises from combining them.

Simple benchmarks. SimplerEnv Bridge contains only four short-horizon pick-and-place tasks. This setting does not test long-horizon, precise, or unseen tasks, where accumulated experience should matter most. Evaluation on a more challenging benchmark, such as RoboDojo, would be much more convincing.

Lack of key ablation. The current ablations do not clearly show that the improvement of the playbook comes from the strong LLM looking at the student LLM’s performance and feedback.

Questions for authors

How does the proposed method compare with existing harness methods, such as Code as Policies and ExpeL?

What new capability comes from combining the three interventions with the playbook, beyond what each existing technique already provides?

Can the method be evaluated on a more challenging benchmark, such as RoboDojo, including long-horizon tasks and tasks not seen during playbook development?

If the strong LLM improves the playbook with the same budget but only from its own rollouts, without seeing the student LLM’s results, how does performance compare with the current refined playbook?

How often is each intervention (instruction, attention, action) used?

Comments on the paper

1 comment

Dear reviewers, the discussion period is coming to an end. Could you please engage with the authors’ rebuttal ASAP? Thanks, AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.