- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes RoboHarness, an agent framework that coordinates existing policies to solve common manipulation tasks in context-dependent scenarios. It combines task understanding, memory, and self-evolution. The framework also introduces Memory Bridge, a policy-chaining algorithm. Memory Bridge learns a local progress ranking from retrieved trajectories and uses it to choose handoff configurations that are reachable and supported by memory. Experiments show large improvements across 500 simulation tasks and 150 real-world trials.
Critical strengths
None
Critical weaknesses
- The results make it hard to tell how much each component contributes. A more complete ablation study is needed. For example, the WAM and RL policies are only available on LIBERO-LoHo and LIBERO-Complex, so it is unclear how much they add. A leave-one-policy-out ablation that also reports how often each policy is invoked would show whether the "heterogeneous" policies actually help, or whether π0.5 + TAMP does most of the work.
- Using TAMP in the loop raises a privileged-information concern, because the agent harness is given ground-truth object states. The authors state that they "inject 5–10% noise into simulator-provided object poses to approximate real-world estimation errors," but they do not explain what 5–10% noise means (e.g., relative to what quantity). The baselines also do not receive object poses, so the comparison is not clean. I cannot tell whether the gains come from the method itself or from this extra information.
- I consider unfair comparison the most critical issue in this work. RoboHarness uses Codex (GPT-5.5) as its task planner, while the single-policy baselines use no task planner at all. In addition, the hierarchical baseline results on LIBERO-LoHo (H-WM-π0.5 84.9/64.8, Logic-guided π0.5 73.2/48.4, LLM-guided π0.5 66.8/26.8) [Huang et al., 2026a] are copied from the H-WM paper, where the planner is a 4B LLM (Qwen3-4B). The authors should redesign the experimental setup so that the comparison is convincing.
- I am concerned about the novelty of this paper. Prior works have already moved the robot to a pose from which the next policy is likely to succeed, either by retrieving stored demonstrations (Johns, 2021; Di Palo & Johns, 2024) or by learning or planning the handoff (Chen et al., 2023; Dalal et al., 2024). None of these are discussed. A nearest-anchor baseline (moving directly to the retrieved pose, as in DINOBot) is needed to show that the ranking-SVM progress estimator actually matters. The self-evolution skills raise a similar concern. Prior LLM-agent and robotics work has already explored updating policy descriptions from past successes and failures, rewriting code from execution errors, and diagnosing failures from execution logs (ExpeL, Zhao et al., 2024; Reflexion, Shinn et al., 2023; Voyager, Wang et al., 2024; REFLECT, Liu et al., 2023; DROC, Zha et al., 2023). The paper does not explain how its evolution skills differ from these methods, so the contribution looks like a combination of existing pieces. If so, the evolution module should be ablated one part at a time to show which parts matter, rather than removed as a single block (Figure 4).
- On RMBench, the baselines either have no memory (DP, π0.5, X-VLA) or have a compact visual memory with an 8B planner (Mem-0). RoboHarness, in contrast, uses a complete execution log read by GPT-5.5, plus simulator object poses (as I read Sec. 5, the pose setting applies to all simulated benchmarks). Because of this, the gains (e.g., Press Button 0 → 100) cannot be attributed to better memory-dependent execution. A fair comparison would give a baseline the same planner and log, and remove the simulator poses from RoboHarness.
- Yuanpei Chen, Chen Wang, Li Fei-Fei, and C. Karen Liu. Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation. CoRL, PMLR 229:3809–3829, 2023.
- Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks. ICLR, 2024.
- Norman Di Palo and Edward Johns. DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models. ICRA, 2024. arXiv:2402.13181.
- Jinbang Huang, Wenyuan Chen, Zhiyuan Li, Oscar Pang, Xiao Hu, Lingfeng Zhang, Yuanzhao Hu, Mark Coates, Tongtong Cao, Xingyue Quan, Zhanguang Zhang, and Yingxue Zhang. H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model. arXiv:2602.11291, 2026.
- Edward Johns. Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration. ICRA, 2021. arXiv:2105.06411. Zeyi Liu, Arpit Bahety, and Shuran Song. REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. CoRL, PMLR 229:3468–3484, 2023.
- Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS, 2023.
- Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. TMLR, 2024.
- Lihan Zha, Yuchen Cui, et al. Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections. arXiv:2311.10678, 2023.
- Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM Agents Are Experiential Learners. AAAI, 38(17):19632–19642, 2024.
Questions for authors
Please see critical waknesses.
5 comments
W2
We address your concerns by clarifying how RoboHarness uses available information, specifying the pose-noise injection procedure, and providing additional evaluations using only visuomotor policies without simulator-provided object poses.
RoboHarness’s ability to use a broader range of available information is itself a system-level advantage. RoboHarness integrates policies with different input requirements into a unified framework, assigning subtasks and invoking policies according to the available information. It can therefore use different information sources through compatible policies without architectural redesign or retraining. Visuomotor policies without pose inputs cannot directly use pose estimates and may require substantial redesign and training to incorporate them.
Noise injection is intended to avoid using exact simulator-provided poses as privileged information. Each position coordinate is perturbed by 5–10% of the corresponding edge length of the object’s bounding box, while each Euler-angle component is perturbed by 5–10% of its full – range. For each component, the percentage and sign are randomly sampled. This perturbation evaluates execution under imperfect pose inputs, approximating the errors that arise when object poses are estimated from visual observations rather than obtained from ground truth. We have clarified this definition in the revised paper (lines 354–357 and 1371–1375).
To address your concern more directly, we evaluate RoboHarness using only visuomotor policies This variant coordinates , Faster-WAM, and OpenVLA-OFT on LIBERO-Plus and LIBERO-LoHo. We compare it against these same policies operating individually, assessing whether RoboHarness provides gains without the additional pose information.
| Method | Robot State | Language | Layout | Background | Sensor | Camera | Light | Overall |
|---|---|---|---|---|---|---|---|---|
| RoboHarness (visuomotor policies only) | 90.0 | 96.8 | 86.5 | 95.2 | 89.7 | 78.8 | 97.6 | 90.1 |
| 77.5 | 85.6 | 85.7 | 94.6 | 89.7 | 75.4 | 96.9 | 85.7 | |
| Faster-WAM | 58.3 | 90.9 | 81.7 | 60.7 | 84.6 | 76.5 | 94.2 | 78.3 |
| OpenVLA-OFT | 21.7 | 81.0 | 68.7 | 91.0 | 78.6 | 55.6 | 92.7 | 67.9 |
Table 2: Success rates (%) on LIBERO-Plus without TAMP or simulator-provided object poses.
On LIBERO-Plus, visuomotor-only RoboHarness achieves 90.1% overall success, outperforming by 4.4 percentage points and matching or exceeding the best individual policy in every perturbation category. Gains are largest in Robot State and Language, consistent with the benefits of Memory Bridge, instruction preprocessing, and dynamic policy assignment.
| Method | SR (%) | PS (%) |
|---|---|---|
| RoboHarness (visuomotor policies only) | 54.8 | 83.2 |
| 6.4 | 55.3 | |
| Faster-WAM | 0.4 | 24.9 |
| OpenVLA-OFT | 0.0 | 20.0 |
Table 3: Success rate (SR) and progress score (PS) on LIBERO-LoHo without TAMP or simulator-provided object poses.
On LIBERO-LoHo, visuomotor-only RoboHarness achieves 54.8% success and 83.2% progress, exceeding by 48.4 and 27.9 percentage points, respectively. These gains support capability coordination through task decomposition, policy assignment, and task-order adjustment. Remaining policy gaps, including unseen grasps across object geometries and locations, limit full-task success despite substantial partial completion.
These evaluations show that RoboHarness provides substantial gains even without simulator-provided object poses. Its advantage over the constituent policies persists on both benchmarks, demonstrating that the improvements cannot be explained solely by access to additional pose information. We have included these results and the clarified noise definition in the revised paper.(Appendices E.1 and line 1371-1375)
W3 + W5
Regarding the comparison of baselines, we provide the following clarification and experiments to address your concerns.
RoboHarness improves what existing heterogeneous policies can accomplish together, rather than what a single policy learns to do. It takes independently developed heterogeneous policies as available capabilities and coordinates them through task decomposition, policy assignment, and handoffs without joint retraining. The contribution therefore lies in how these capabilities are combined into a functioning system. Our experiments are designed to evaluate whether orchestration yields system-level gains, rather than the quality of a newly trained policy under an otherwise identical architecture. This clarification has been added to the revised paper in line 66-69.
Consistent with this system-level objective, our evaluation compares complete systems under a common benchmark protocol. Shared benchmarks allow us to evaluate policy orchestration against published results without reimplementing and rerunning every prior method. Matching components is not a simple substitution, because each method relies on its own architecture and training procedure. H-WM [1] fine-tunes its planner, world model, and VLA on benchmark data, with execution relying on aligned logical and visual guidance between modules; replacing the planner alone would disrupt the alignment learned among these components. RoboHarness reads execution logs generated by its own orchestration process, which Mem-0 does not. Mem-0’s planner uses key-frame memory, while its execution module uses anchor and sliding-window memories. Supplying RoboHarness execution logs would therefore require adapting its memory interfaces and validating whether its planner can effectively use this additional information. Adding memory mechanisms to VLAs would likewise create new methods, with development and training costs beyond the scope of this comparison.
To directly assess whether the gains primarily come from a stronger planner, we provide an additional LIBERO-LoHo baseline using the same GPT-5.5 guided VLA backbone. Specifically, we use GPT-5.5 to decompose tasks and guide execution. This baseline achieves an SR of 28.8% and a PS of 71.2%, comparable to the Qwen-based LLM-guided results reported in the paper (SR 26.8%, PS 66.8%), suggesting that planner capability is not the primary bottleneck in this setting. These results support that RoboHarness’s gains stem from heterogeneous policy orchestration rather than a stronger planner alone.
Object poses do not resolve the memory requirements of Press Button. The robot must track which buttons have been successfully pressed and how many times, according to two digit tiles, before pressing the right button to confirm. Object poses do not provide these accumulated counts. To further assess performance without additional pose information, we evaluate visuomotor-only RoboHarness on RMBench. After removing TAMP and simulator-provided object poses from the original setting, RoboHarness achieves 61.7% overall success, exceeding Mem-0 by 19.7 percentage points (Table 4).
| Method | SR (%) | SR |
|---|---|---|
| RoboHarness (visuomotor policies only) | 61.7 | 0.0 |
| Mem-0 | 42.0 | -19.7 |
| 10.4 | -51.3 | |
| X-VLA | 9.8 | -51.9 |
| Diffusion Policy | 5.8 | -55.9 |
Table 4: Success rate (SR) on RMBench. Baseline results are from the RMBench paper. Deltas are relative to the visuomotor-only RoboHarness variant, in percentage points.
We have included the two additional experiments in (Appendix E.6) and (Appendix E.1, TABLE 12)
Thank you for your detailed feedback. We have provided additional analysis and baseline experiments to address your concerns. We hope these additions help clarify our method and the paper’s positioning and support further discussion.
We would first like to clarify the paper’s positioning: RoboHarness is an orchestration system that coordinates independently developed heterogeneous policies without joint training, combining their strengths and compensating for their weaknesses to accomplish tasks beyond any individual policy’s capabilities. Achieving this requires addressing two central bottlenecks: (1) capability-aware task decomposition and subtask assignment, and (2) reliable handoffs between heterogeneous policies. Our design addresses these bottlenecks, and our experiments evaluate whether the resulting orchestration improves system-level performance. This system-level objective, rather than training a stronger standalone policy, guides our responses to your concerns below.
W1
We clarify the benchmark-specific policy configurations and provide additional ablations and policy usage statistics to explain how the WAM and RL policies contribute to the full system.
Our framework can coordinate varying numbers of heterogeneous policies without joint training. The policy library can contain two, four, or even more policies, depending on the availability of compatible implementations and checkpoints. Below, we address you concerns in detail.
Contributions of WAM and RL Policies. To assess the contributions of the WAM and RL policies, we remove each separately from the full policy library and repeat the evaluation on LIBERO-Complex, reporting both performance and policy usage.
Removing either Faster-WAM or RL-OpenVLA decreases performance, indicating that both contribute to the full system, although their contributions differ in magnitude.
The smaller drop from removing Faster-WAM is consistent with its training coverage overlapping with that of π0.5. We inspected their code repositories and found that both use LIBERO-10, LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal for training/post-training. This shared coverage helps explain why π0.5 can compensate for much of the capability lost when Faster-WAM is removed. A separate ablation removing π0.5 also yields a relatively small drop (SR: 83.2%; PS: 92.2%), supporting this interpretation of overlapping capabilities. Nevertheless, their execution success rates differ on individual tasks, allowing RoboHarness to favor the more reliable policy and achieve higher overall performance when both are available.
Removing RL-OpenVLA has the largest impact. RL-OpenVLA is post-trained on LIBERO-90, which provides substantially different task coverage. This helps explain the larger capability gap after its removal and the difficulty of compensating with the remaining policies.
Table 1: Policy-removal ablations on LIBERO-Complex. Policy usage is the percentage of evaluation instances in which a policy is invoked at least once; multiple policies may be invoked within an instance. Deltas are relative to the full four-policy library, in percentage points. RL, VLA, and WAM denote RL-OpenVLA, π0.5, and Faster-WAM, respectively.
We have included the above discussion in appendix of the revised paper.(Appendix E.4)