- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes the Recursive Harness Distillation (RHD) framework to improve the execution reliability of vision-language-action (VLA) models in robotic manipulation tasks. The core idea is to enable a strong agent to distill its execution experience into reusable Playbooks and recursively refine them based on feedback from a lightweight agent. These Playbooks improve VLA performance through instruction, attention, and action interventions without updating model parameters.
Experiments are conducted in SimplerEnv Bridge and on real robots. In simulation, the task success rate of the lightweight agent increases from 43.8% to 66.7% with the optimized Playbooks. On real robots, the success rate improves from 37.3% to 64.0%. Ablation studies further demonstrate the importance of different intervention methods, recursive refinement, and guidance from a stronger model. These results suggest that the proposed framework effectively transfers execution experience across agents.
Critical strengths
S1. The method offers practical value. The paper proposes transferring experience across agents through recursive Playbook refinement without updating model parameters. This provides a low-cost and easy-to-deploy approach to improving the execution capabilities of existing VLA models.
S2. The experimental results are convincing. The method achieves significant improvements in both simulation and real-world robotic tasks, reaching success rates of 66.7% and 64.0%, respectively. These results demonstrate the effectiveness of the framework in simulation and its potential for real-world applications.
S3. The ablation studies are relatively comprehensive. The paper examines the contributions of recursive refinement, teacher model capabilities, and different intervention methods. It also compares deployment costs. These experiments help explain the sources of performance improvements and support the design choice of using a stronger model to guide a lightweight agent.
Critical weaknesses
W1. Insufficient baseline comparisons. Although the paper compares initial and optimized Playbooks, it lacks direct comparisons with methods such as non-recursive experience summarization and traditional prompt optimization. Therefore, it is difficult to determine how much of the performance improvement comes from the recursive distillation mechanism itself.
W2. Limited evaluation of generalization. The experiments cover only four simulation tasks and three real-world robotic tasks. Moreover, the Playbooks require task-specific adaptation in real-world settings. It remains unclear whether the method can effectively generalize to different robots, VLA models, and more complex tasks.
W3. Unclear practical costs and stability. The paper mainly compares inference costs during deployment but does not fully quantify the total cost of strong model exploration and multiple rounds of Playbook refinement. In addition, the initial Playbooks actually reduce the success rate, suggesting that the method may be sensitive to the quality of the distilled experience and the refinement process.
Questions for authors
Q1. Have the authors considered comparisons with other Playbook optimization methods, such as one-time experience summarization or simple prompt optimization? How can they demonstrate that the performance gains mainly come from recursive distillation rather than additional experience accumulation?
Q2. How many iterations does recursive Playbook refinement require to converge? Are there significant differences in convergence speed and final performance across tasks?
Q3. In the real-world robot experiments, the Playbooks require task-specific adaptation by a stronger model. How much additional experience does this adaptation require? How much of the final performance improvement can be attributed to knowledge transferred from the original Playbooks?
Q4. The paper mainly compares inference costs during deployment. Could the authors also report the total costs of strong model exploration, Playbook generation, and multiple refinement rounds to better assess the practical cost-effectiveness of the method?
Q5. The initial Playbook reduces the success rate from 43.8% to 31.3%. Could the authors explain the main reasons for this performance drop and how the recursive refinement process prevents incorrect experiences from causing further performance degradation?
Thanks to the authors for the detailed responses and additional experiments. Most of my concerns have been addressed. However, it is still unclear how much of the improvement comes from recursive refinement itself rather than simply using more execution experience. The total development cost is also not fully reported, and the stability of the refinement process remains a concern. Overall, I appreciate the additional results, but I will keep my rating unchanged at Weak Acceptance.