acceptodds
Under review as a conference paper at ICLR 2027

RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning

Abstract

Long-horizon robotic tasks require a breadth of capabilities beyond what any single existing robot control policy can reliably provide. Combining heterogeneous policies with complementary strengths offers a promising solution, but introduces two key challenges: uncertain capability boundaries and distribution mismatches during policy handoffs. These challenges remain largely unaddressed by existing planning methods, which typically assume homogeneous, predefined skills with fixed applicability. We propose RoboHarness, a unified framework that encapsulates independently developed heterogeneous policies, including vision-language-action models (VLAs), world-action models (WAMs), reinforcement learning (RL) policies, and task and motion planners (TAMP), as reusable agentic skills. RoboHarness integrates understanding, memory, and evolution skills to reason about policy capabilities and support capability-aware task decomposition and policy routing. To mitigate distribution mismatches during policy handoffs, we introduce Memory Bridge, a plug-in policy-chaining mechanism that enables reliable transitions between heterogeneous policies without joint retraining. Extensive experiments across five public benchmarks, 500 customized tasks across 10 classes, and 135 real-robot trials demonstrate substantial gains in long-horizon and memory-dependent tasks, as well as robustness to out-of-distribution conditions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Agent System, Task and Motion Planning, Long-horizon planning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_kAffOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes RoboHarness, an agent framework that coordinates existing policies to solve common manipulation tasks in context-dependent scenarios. It combines task understanding, memory, and self-evolution. The framework also introduces Memory Bridge, a policy-chaining algorithm. Memory Bridge learns a local progress ranking from retrieved trajectories and uses it to choose handoff configurations that are reachable and supported by memory. Experiments show large improvements across 500 simulation tasks and 150 real-world trials.

Critical strengths

None

Critical weaknesses
  1. The results make it hard to tell how much each component contributes. A more complete ablation study is needed. For example, the WAM and RL policies are only available on LIBERO-LoHo and LIBERO-Complex, so it is unclear how much they add. A leave-one-policy-out ablation that also reports how often each policy is invoked would show whether the "heterogeneous" policies actually help, or whether π0.5 + TAMP does most of the work.
  2. Using TAMP in the loop raises a privileged-information concern, because the agent harness is given ground-truth object states. The authors state that they "inject 5–10% noise into simulator-provided object poses to approximate real-world estimation errors," but they do not explain what 5–10% noise means (e.g., relative to what quantity). The baselines also do not receive object poses, so the comparison is not clean. I cannot tell whether the gains come from the method itself or from this extra information.
  3. I consider unfair comparison the most critical issue in this work. RoboHarness uses Codex (GPT-5.5) as its task planner, while the single-policy baselines use no task planner at all. In addition, the hierarchical baseline results on LIBERO-LoHo (H-WM-π0.5 84.9/64.8, Logic-guided π0.5 73.2/48.4, LLM-guided π0.5 66.8/26.8) [Huang et al., 2026a] are copied from the H-WM paper, where the planner is a 4B LLM (Qwen3-4B). The authors should redesign the experimental setup so that the comparison is convincing.
  4. I am concerned about the novelty of this paper. Prior works have already moved the robot to a pose from which the next policy is likely to succeed, either by retrieving stored demonstrations (Johns, 2021; Di Palo & Johns, 2024) or by learning or planning the handoff (Chen et al., 2023; Dalal et al., 2024). None of these are discussed. A nearest-anchor baseline (moving directly to the retrieved pose, as in DINOBot) is needed to show that the ranking-SVM progress estimator actually matters. The self-evolution skills raise a similar concern. Prior LLM-agent and robotics work has already explored updating policy descriptions from past successes and failures, rewriting code from execution errors, and diagnosing failures from execution logs (ExpeL, Zhao et al., 2024; Reflexion, Shinn et al., 2023; Voyager, Wang et al., 2024; REFLECT, Liu et al., 2023; DROC, Zha et al., 2023). The paper does not explain how its evolution skills differ from these methods, so the contribution looks like a combination of existing pieces. If so, the evolution module should be ablated one part at a time to show which parts matter, rather than removed as a single block (Figure 4).
  5. On RMBench, the baselines either have no memory (DP, π0.5, X-VLA) or have a compact visual memory with an 8B planner (Mem-0). RoboHarness, in contrast, uses a complete execution log read by GPT-5.5, plus simulator object poses (as I read Sec. 5, the pose setting applies to all simulated benchmarks). Because of this, the gains (e.g., Press Button 0 → 100) cannot be attributed to better memory-dependent execution. A fair comparison would give a baseline the same planner and log, and remove the simulator poses from RoboHarness.
  • Yuanpei Chen, Chen Wang, Li Fei-Fei, and C. Karen Liu. Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation. CoRL, PMLR 229:3809–3829, 2023.
  • Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks. ICLR, 2024.
  • Norman Di Palo and Edward Johns. DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models. ICRA, 2024. arXiv:2402.13181.
  • Jinbang Huang, Wenyuan Chen, Zhiyuan Li, Oscar Pang, Xiao Hu, Lingfeng Zhang, Yuanzhao Hu, Mark Coates, Tongtong Cao, Xingyue Quan, Zhanguang Zhang, and Yingxue Zhang. H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model. arXiv:2602.11291, 2026.
  • Edward Johns. Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration. ICRA, 2021. arXiv:2105.06411. Zeyi Liu, Arpit Bahety, and Shuran Song. REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. CoRL, PMLR 229:3468–3484, 2023.
  • Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS, 2023.
  • Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. TMLR, 2024.
  • Lihan Zha, Yuchen Cui, et al. Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections. arXiv:2311.10678, 2023.
  • Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM Agents Are Experiential Learners. AAAI, 38(17):19632–19642, 2024.
Questions for authors

Please see critical waknesses.

5 comments

Thank you for your detailed feedback. We have provided additional analysis and baseline experiments to address your concerns. We hope these additions help clarify our method and the paper’s positioning and support further discussion.

We would first like to clarify the paper’s positioning: RoboHarness is an orchestration system that coordinates independently developed heterogeneous policies without joint training, combining their strengths and compensating for their weaknesses to accomplish tasks beyond any individual policy’s capabilities. Achieving this requires addressing two central bottlenecks: (1) capability-aware task decomposition and subtask assignment, and (2) reliable handoffs between heterogeneous policies. Our design addresses these bottlenecks, and our experiments evaluate whether the resulting orchestration improves system-level performance. This system-level objective, rather than training a stronger standalone policy, guides our responses to your concerns below.

W1

We clarify the benchmark-specific policy configurations and provide additional ablations and policy usage statistics to explain how the WAM and RL policies contribute to the full system.

Our framework can coordinate varying numbers of heterogeneous policies without joint training. The policy library can contain two, four, or even more policies, depending on the availability of compatible implementations and checkpoints. Below, we address you concerns in detail.

Contributions of WAM and RL Policies. To assess the contributions of the WAM and RL policies, we remove each separately from the full policy library and repeat the evaluation on LIBERO-Complex, reporting both performance and policy usage.

Removing either Faster-WAM or RL-OpenVLA decreases performance, indicating that both contribute to the full system, although their contributions differ in magnitude.

The smaller drop from removing Faster-WAM is consistent with its training coverage overlapping with that of π0.5\pi_{0.5}. We inspected their code repositories and found that both use LIBERO-10, LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal for training/post-training. This shared coverage helps explain why π0.5\pi_{0.5} can compensate for much of the capability lost when Faster-WAM is removed. A separate ablation removing π0.5\pi_{0.5} also yields a relatively small drop (SR: 83.2%; PS: 92.2%), supporting this interpretation of overlapping capabilities. Nevertheless, their execution success rates differ on individual tasks, allowing RoboHarness to favor the more reliable policy and achieve higher overall performance when both are available.

Removing RL-OpenVLA has the largest impact. RL-OpenVLA is post-trained on LIBERO-90, which provides substantially different task coverage. This helps explain the larger capability gap after its removal and the difficulty of compensating with the remaining policies.

ConfigurationSR (%)PS (%)Δ\DeltaSR / Δ\DeltaPSTAMP usage (%)RL usage (%)VLA usage (%)WAM usage (%)
Full four-policy library86.894.2–98.442.827.416.8
Without RL-OpenVLA56.077.0-30.8 / -17.298.8–18.222.2
Without Faster-WAM85.293.0-1.6 / -1.299.240.030.8–

Table 1: Policy-removal ablations on LIBERO-Complex. Policy usage is the percentage of evaluation instances in which a policy is invoked at least once; multiple policies may be invoked within an instance. Deltas are relative to the full four-policy library, in percentage points. RL, VLA, and WAM denote RL-OpenVLA, π0.5\pi_{0.5}, and Faster-WAM, respectively.

We have included the above discussion in appendix of the revised paper.(Appendix E.4)

W2

We address your concerns by clarifying how RoboHarness uses available information, specifying the pose-noise injection procedure, and providing additional evaluations using only visuomotor policies without simulator-provided object poses.

RoboHarness’s ability to use a broader range of available information is itself a system-level advantage. RoboHarness integrates policies with different input requirements into a unified framework, assigning subtasks and invoking policies according to the available information. It can therefore use different information sources through compatible policies without architectural redesign or retraining. Visuomotor policies without pose inputs cannot directly use pose estimates and may require substantial redesign and training to incorporate them.

Noise injection is intended to avoid using exact simulator-provided poses as privileged information. Each position coordinate is perturbed by 5–10% of the corresponding edge length of the object’s bounding box, while each Euler-angle component is perturbed by 5–10% of its full 00–2π2\pi range. For each component, the percentage and sign are randomly sampled. This perturbation evaluates execution under imperfect pose inputs, approximating the errors that arise when object poses are estimated from visual observations rather than obtained from ground truth. We have clarified this definition in the revised paper (lines 354–357 and 1371–1375).

To address your concern more directly, we evaluate RoboHarness using only visuomotor policies This variant coordinates π0.5\pi_{0.5}, Faster-WAM, and OpenVLA-OFT on LIBERO-Plus and LIBERO-LoHo. We compare it against these same policies operating individually, assessing whether RoboHarness provides gains without the additional pose information.

MethodRobot StateLanguageLayoutBackgroundSensorCameraLightOverall
RoboHarness (visuomotor policies only)90.096.886.595.289.778.897.690.1
π0.5\pi_{0.5}77.585.685.794.689.775.496.985.7
Faster-WAM58.390.981.760.784.676.594.278.3
OpenVLA-OFT21.781.068.791.078.655.692.767.9

Table 2: Success rates (%) on LIBERO-Plus without TAMP or simulator-provided object poses.

On LIBERO-Plus, visuomotor-only RoboHarness achieves 90.1% overall success, outperforming π0.5\pi_{0.5} by 4.4 percentage points and matching or exceeding the best individual policy in every perturbation category. Gains are largest in Robot State and Language, consistent with the benefits of Memory Bridge, instruction preprocessing, and dynamic policy assignment.

MethodSR (%)PS (%)
RoboHarness (visuomotor policies only)54.883.2
π0.5\pi_{0.5}6.455.3
Faster-WAM0.424.9
OpenVLA-OFT0.020.0

Table 3: Success rate (SR) and progress score (PS) on LIBERO-LoHo without TAMP or simulator-provided object poses.

On LIBERO-LoHo, visuomotor-only RoboHarness achieves 54.8% success and 83.2% progress, exceeding π0.5\pi_{0.5} by 48.4 and 27.9 percentage points, respectively. These gains support capability coordination through task decomposition, policy assignment, and task-order adjustment. Remaining policy gaps, including unseen grasps across object geometries and locations, limit full-task success despite substantial partial completion.

These evaluations show that RoboHarness provides substantial gains even without simulator-provided object poses. Its advantage over the constituent policies persists on both benchmarks, demonstrating that the improvements cannot be explained solely by access to additional pose information. We have included these results and the clarified noise definition in the revised paper.(Appendices E.1 and line 1371-1375)

W3 + W5

Regarding the comparison of baselines, we provide the following clarification and experiments to address your concerns.

RoboHarness improves what existing heterogeneous policies can accomplish together, rather than what a single policy learns to do. It takes independently developed heterogeneous policies as available capabilities and coordinates them through task decomposition, policy assignment, and handoffs without joint retraining. The contribution therefore lies in how these capabilities are combined into a functioning system. Our experiments are designed to evaluate whether orchestration yields system-level gains, rather than the quality of a newly trained policy under an otherwise identical architecture. This clarification has been added to the revised paper in line 66-69.

Consistent with this system-level objective, our evaluation compares complete systems under a common benchmark protocol. Shared benchmarks allow us to evaluate policy orchestration against published results without reimplementing and rerunning every prior method. Matching components is not a simple substitution, because each method relies on its own architecture and training procedure. H-WM [1] fine-tunes its planner, world model, and VLA on benchmark data, with execution relying on aligned logical and visual guidance between modules; replacing the planner alone would disrupt the alignment learned among these components. RoboHarness reads execution logs generated by its own orchestration process, which Mem-0 does not. Mem-0’s planner uses key-frame memory, while its execution module uses anchor and sliding-window memories. Supplying RoboHarness execution logs would therefore require adapting its memory interfaces and validating whether its planner can effectively use this additional information. Adding memory mechanisms to VLAs would likewise create new methods, with development and training costs beyond the scope of this comparison.

To directly assess whether the gains primarily come from a stronger planner, we provide an additional LIBERO-LoHo baseline using the same GPT-5.5 guided VLA backbone. Specifically, we use GPT-5.5 to decompose tasks and guide π0.5\pi_{0.5} execution. This baseline achieves an SR of 28.8% and a PS of 71.2%, comparable to the Qwen-based LLM-guided π0.5\pi_{0.5} results reported in the paper (SR 26.8%, PS 66.8%), suggesting that planner capability is not the primary bottleneck in this setting. These results support that RoboHarness’s gains stem from heterogeneous policy orchestration rather than a stronger planner alone.

Object poses do not resolve the memory requirements of Press Button. The robot must track which buttons have been successfully pressed and how many times, according to two digit tiles, before pressing the right button to confirm. Object poses do not provide these accumulated counts. To further assess performance without additional pose information, we evaluate visuomotor-only RoboHarness on RMBench. After removing TAMP and simulator-provided object poses from the original setting, RoboHarness achieves 61.7% overall success, exceeding Mem-0 by 19.7 percentage points (Table 4).

MethodSR (%)Δ\DeltaSR
RoboHarness (visuomotor policies only)61.70.0
Mem-042.0-19.7
π0.5\pi_{0.5}10.4-51.3
X-VLA9.8-51.9
Diffusion Policy5.8-55.9

Table 4: Success rate (SR) on RMBench. Baseline results are from the RMBench paper. Deltas are relative to the visuomotor-only RoboHarness variant, in percentage points.

We have included the two additional experiments in (Appendix E.6) and (Appendix E.1, TABLE 12)

Reviewer_Vp61OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboHarness addresses long-horizon robotic tasks by coordinating independently developed, heterogeneous policies. It combines capability assessment, episodic memory, and online evolution for task decomposition and policy selection. Memory Bridge retrieves successful trajectories of the incoming policy and uses a local progress-ranking model to choose a reachable handoff configuration. The system is evaluated on manipulation benchmarks, customized tasks, and real robots, with memory and system updates retained across trials of the same task.

Critical strengths
  1. The paper treats policy selection and policy handoff as separate problems. This is a useful distinction: selecting a capable policy is not enough if the preceding policy leaves the robot in a configuration from which it cannot execute reliably.
  2. The system has broad experimental support, including long-horizon tasks and 135 real-robot trials. On LIBERO-Complex, removing Memory Bridge reduces success from 86.8% to 60.8%, indicating that handoff handling contributes substantially to the full system.
Critical weaknesses
  1. The experiments use a GPT-5.5-powered coding agent. I would like to see how much RoboHarness adds when the backbone is GPT-6 Astra. The recent Astra embodied-policy study evaluates Direct numerical action generation and Hybrid control with a learned policy [1], making these concrete baselines to consider. The key comparison is Astra with and without RoboHarness, using the same policy library where applicable; Direct or Hybrid control can provide additional references where the action interface permits. Observations, initial memory, adaptation opportunities, execution budgets, and inference latency need to be matched or reported explicitly. Without this comparison, it is difficult to separate the contribution of the harness from that of the reasoning backbone.

[1] Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies. arXiv:2609.38537. https://arxiv.org/abs/2609.38537

  1. The drop from 86.8% to 60.8% in Figure 4 shows that Memory Bridge matters, but removing the entire bridge also removes retrieval and motion planning. It does not tell us whether the learned progress ranking is necessary. Relevant controls are a neutral-pose transition, the nearest feasible retrieved configuration, and a ranking-free bridge, with the same memory and planning budget. Success of the incoming policy after each handoff, transition cost, and end-to-end success should be reported for these variants.

  2. RMBench includes dual-arm tasks, but aggregate success rates give little information about tightly coupled bimanual coordination. Does Memory Bridge select a joint two-arm configuration subject to inter-arm collision and task constraints, or are the arms handled separately? A synchronized manipulation or handover experiment belongs in the main evaluation, with ablations of task allocation and handoff handling. This would test whether the system can enforce coordination constraints beyond sequencing individual skills.

  3. Appendix A.1 lists ten templates and 500 randomized instances. Were the configurations used to design or tune the harness separate from those used for evaluation? The statement that approximately 20% of subtasks can be solved by only one policy type also needs a clearer basis. Was this determined by explicit capability restrictions or by measured success rates of the selected checkpoints? Per-policy subtask success rates would make this distinction easier to assess.

Questions for authors

Please see the weaknesses.

5 comments

We sincerely thank you for your thoughtful review and positive assessment of our work. We appreciate your recognition of the distinction between policy selection and handoff, as well as the experimental evidence supporting Memory Bridge’s contribution to reliable execution. Your constructive suggestions have helped us strengthen our evaluation through additional experiments and clarifications, which we present below.

W1

We were also excited by Astra’s potential and investigated how its capabilities translate into robot performance when used as the backbone of RoboHarness. We address your concerns through additional experiments comparing Astra with and without RoboHarness. The experiments and discussion are included in Appendix E.2 of the revised paper.

We compared GPT-6 Astra with and without RoboHarness on the real-robot Build Bridge task. For Astra + RoboHarness, we followed the original experimental protocol and changed only the coding-agent backbone. For Astra alone, we provided the same camera observations and low-level robot control API, along with demonstration trajectories from the memory bank as references, but no RoboHarness policies or skills. Astra could interactively write, execute, and revise its own perception, planning, and control code, providing greater flexibility than direct numerical action generation [1]. We evaluated this setting under both the original and an extended time budget.

SettingTime limitSuccess (%)
GPT-5.5 + RoboHarness (paper)10 min86.7
Astra + RoboHarness10 min100.0
Astra alone10 min0.0
Astra aloneExtended100.0

Table 1: Backbone and harness comparison on the real-robot Build Bridge task. .

RoboHarness shifts the coding agent’s workload from building robot capabilities to orchestrating available ones. Without the harness, Astra spends substantial time developing and debugging perception, planning, and control code, typically exhausting the original time budget before the robot starts moving. Its eventual success with additional time indicates that the main bottleneck in this setting is constructing an executable solution within the available budget. RoboHarness supplies reusable policies and supporting skills, allowing Astra to focus on task decomposition, policy selection, and failure recovery.

A stronger backbone complements the harness rather than replacing its role. Astra improves success over GPT-5.5 within RoboHarness, and we also observed stronger recovery from previously unseen failures during task execution. Meanwhile, Astra alone requires substantially more time. These results suggest that stronger backbone reasoning improves orchestration and failure recovery, while the harness provides reusable robot capabilities that would otherwise need to be constructed during execution.

W2

We conducted additional real-robot experiments to isolate the contribution of progress ranking and include the full discussion below in the revised paper (Appendix E.3).

We use the same setup as the real-robot experiments in the paper and evaluate two handoff points: from motion-planner-based object placement to VLA-based cabinet drawer opening or closing. We replace only the Memory Bridge at these handoffs, keeping the remaining execution pipeline unchanged. Each method is evaluated on 20 opening and 20 closing handoffs. All variants use the same RRT motion planner implemented through OMPL.

We measure success over the complete sequence from bridging through completion of the subsequent drawer-opening or drawer-closing subtask, together with the computation time required to generate the bridge trajectory. All methods have a 150 ms planning budget: a trial is successful only if bridge planning finishes within this budget and the subsequent subtask succeeds. For trials exceeding the budget, we record failure but allow planning to finish to measure its computation time. Thus, the reported mean planning time includes over-budget trials.

We compare four variants: (1) Home pose, which returns the robot to its initial configuration before invoking the incoming VLA; (2) Directly move to closest retrieved pose, which plans directly to a retrieved configuration; (3) Memory Bridge without ranking, which retains the memory-supported confidence region but removes progress ranking; and (4) Full Memory Bridge. The memory-based variants use the same memory.

MethodHandoff SRMean planning time
Home pose0%62.1 ms
Directly move to closest retrieved pose52.5%647.5 ms
Memory Bridge without ranking55%175.7 ms
Memory Bridge90%110.2 ms

Table 2: Handoff success under a 150 ms planning budget and mean bridge-planning time.

A neutral pose does not ensure compatibility with the incoming policy. Returning home is computationally inexpensive, but places the robot outside the VLA’s effective starting-state distribution for cabinet manipulation, causing subsequent execution to fail.

Directly moving to retrieved configurations cannot resolve mismatches with the current scene. During cabinet closing, we observed that the memory retrieval module selected robot configurations from partially closed-drawer states that were in collision with the fully open drawer in the current scene. Frames at different stages of drawer closure are visually similar, making it difficult for retrieval alone to identify the exact match to the current drawer state. Planning to these exact configurations repeatedly failed, increasing computation time. The ranking-free bridge reduced this overhead by bounding the sampling region, but still encountered similar failures because it lacked progress-based guidance for selecting alternative handoff targets.

Progress ranking enables flexible, directed handoff planning. Full Memory Bridge uses temporal neighborhoods around retrieved anchors to estimate local progress, allowing nearby high-scoring configurations instead of requiring an exact retrieved target. Combined with the motion-cost penalty evaluated in the current scene, this heuristic guides the search toward reachable, policy-compatible handoff states, improving success while reducing planning overhead.

W3

We clarify below how Memory Bridge handles the two arms across benchmarks. Experimental details are included in the revised paper.

Sequential processing on RMBench: Memory Bridge handles each arm separately when the task permits one-arm-at-a-time execution. Although RMBench runs on the dual-arm AgileX platform, none of the nine evaluated tasks requires the two arms to manipulate the same object or transfer an object between them. RoboHarness therefore executes these tasks by moving one arm at a time. Memory Bridge selects the handoff configuration for the active arm while the inactive arm remains at a fixed configuration. This avoids jointly estimating progress over both arms, but does not evaluate synchronized bimanual coordination. The RMBench results should therefore be interpreted as evidence of memory-dependent execution.

Joint processing on BEHAVIOR1K: Memory Bridge selects a handoff configuration for both arms simultaneously. The most direct extension of the original Memory Bridge formulation takes the end-effector poses of both grippers as simultaneous inputs, together with the robot’s full-body joint configuration. This is the formulation used on BEHAVIOR-1K. However, increasing input dimensions can make local progress estimation more difficult. The evaluated BEHAVIOR-1K tasks include coordinated two-arm manipulation. For example, in Turning On Radio, one arm holds the radio while the other presses a button on it, requiring a handoff configuration that places both arms close together without inter-arm collision. RoboHarness achieves a success rate of 0.90, compared with at most 0.30 for the baselines (Appendix D.3). These results demonstrate execution involving coordinated use of both arms.

To compare joint and separate-arm handoff handling, we evaluated a sequential Memory Bridge variant that handles each arm separately over 10 trials on BEHAVIOR-1K’s Turning On Radio.

Memory Bridge settingSRPS
Joint two-arm0.900.9
Sequential separate-arm0.800.80

The two settings achieve similar success rates. The additional failure in the sequential variant occurred when the VLA failed to maintain a stable grasp on the radio, rather than during the bridge transition. This limited difference is consistent with Memory Bridge’s role: it guides the robot into a suitable starting region for the incoming policy, while the policy handles the precise grasping and button-pressing motions.

In the revision, we have distinguished these two settings explicitly, clarify which benchmark evaluates coordinated two-arm execution, and extend the state definition to include both arms and discuss the associated limitations.(Appendix C.3)

W4

Yes, the configurations used to develop the harness were entirely separate from those used for evaluation. RoboHarness uses a unified framework, without task- or environment-specific tuning before evaluation. Any online adaptation during evaluation follows the same mechanisms described in the paper.

The original statement was based on inspection of each policy’s implementation during task design; we agree that empirical statistics provide a clearer basis for assessing policy coverage. We therefore evaluated each policy at the subtask level. The table below reports the percentage of subtasks on which each policy achieved zero successes across all evaluated trials.

PolicyZero-success subtask ratio (%)
Motion planner20.59
π0.5\pi_{0.5}41.18
RLinf OpenVLA85.29
FasterWAM58.82

Table 3: Per-policy zero-success subtask coverage.

These results show that no single evaluated policy covers the full set of subtasks. Even the policy with the broadest coverage requires other policies to handle at least 20% of subtasks, supporting the need for capability-aware orchestration. We have added these statistics to (lines 874–879) and replace the original statement with this result-based description, explicitly grounding it in the observed performance of the evaluated implementations and checkpoints.

References

[1] Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies. arXiv:2609.38537. https://arxiv.org/abs/2609.38537

Thank you again for taking the time to review our work and share your suggestions. We hope the additional experiments and clarifications help resolve your concerns, and we would be glad to discuss any remaining questions.

Reviewer_QkhVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RoboHarness, an agentic orchestration framework that coordinates heterogeneous robot control policies (VLAs, WAMs, RL policies, and TAMP) via capability-aware task decomposition, online experience evolution, and a Memory Bridge module that aligns inter-policy handoff states without joint retraining.

Critical strengths
  • Addressing distribution mismatches and uncertain capability boundaries when chaining independently trained, heterogeneous robot policies (combining VLAs, WAMs, RL, and TAMP) is a highly practical and timely systems contribution for long-horizon manipulation.
  • The Memory Bridge mechanism—which retrieves successful trajectories from the incoming policy's memory bank and fits a local ranking SVM to guide motion-planned handoff transitions—is a clever, lightweight solution that avoids expensive joint policy finetuning.
  • The empirical evaluation is extensive, spanning five public simulation benchmarks (LIBERO, LIBERO-Plus, LIBERO-LoHo, RMBench, and BEHAVIOR-1K), 500 customized LIBERO-Complex tasks, and 135 real-world UR5e assembly trials under various physical and perceptual disturbances.
Critical weaknesses
  • Privileged State Assumptions and Unfair Baseline Comparisons: RoboHarness includes an explicit TAMP planner that consumes simulator-provided object poses (with 5-10% injected noise) or ArUco marker poses in the real world, whereas the end-to-end VLA and WAM baselines in Tables 1 and 2 operate purely from raw visual observations and proprioception. This creates an inherent informational advantage, and in several benchmarks (LIBERO, LIBERO-Plus, RMBench, and BEHAVIOR-1K in Table 3), the system only orchestrates Pi0.5 and TAMP rather than the full four-family heterogeneous library.
  • Online Experience Accumulation Across Evaluation Episodes: During benchmark evaluation, RoboHarness continuously updates its execution log, refines orchestration code and hyperparameters via the evolution module, and appends successful trajectories to its episodic memory across trials of the same task. Comparing a test-time adapting and code-evolving system against static, zero-shot evaluated baselines obscures how much of the gain comes from initial orchestration versus multi-episode test-time learning.
  • Purely Proprioceptive Formulation of Memory Bridge: The local progress estimator in Memory Bridge fits a linear ranking SVM exclusively on normalized robot joint angles and end-effector poses without conditioning on dynamic object states in the workspace. While effective when retrieved anchors share near-identical visual layouts, a purely robot-state-based handoff target can become invalid or cause collisions if target objects have shifted relative to the retrieved demonstration.
  • Substantial Inference Latency and Dependency on Proprietary Coding Agents: The orchestration relies heavily on repeated API calls to a heavy coding agent (Codex with GPT-5.5), which incurs an average latency of 4.6 seconds per call and adds 67.4 seconds of overhead per trial in real-robot experiments, making timeouts the primary failure mode and limiting applicability in dynamic or time-critical tasks.
Questions for authors
  1. To ensure a fair comparison with vision-only baselines, what is the performance of RoboHarness on LIBERO-Plus, LIBERO-LoHo, and RMBench when the TAMP planner (which relies on object pose estimation) is excluded and only learned visuomotor policies (Pi0.5, Faster-WAM, RLinf-OpenVLA) are orchestrated?
  2. What is the first-episode (zero-shot) success rate on the benchmarks when online memory accumulation and evolution updates across evaluation trials of the same task are completely frozen?
  3. How does Memory Bridge avoid invalid handoff configurations when the target object's spatial position in the current scene differs noticeably from its position in the top-K retrieved memory trajectories, given that the ranking SVM only scores robot proprioceptive states?
  4. How sensitive is RoboHarness to the choice of the backbone coding agent (e.g., replacing GPT-5.5 with smaller open-weight models to reduce the 67.4s per-trial reasoning latency)?
5 comments

Thank you for your thoughtful review and encouraging assessment. We appreciate your recognition of heterogeneous policy orchestration’s practical value, Memory Bridge’s lightweight design, and our broad evaluation. Your suggestions helped strengthen our evidence and clarify the system design. Below, we address your concerns with additional experiments and analysis.

W1 + Q1

We address your concerns about privileged state assumptions and baseline comparisons with clarifications, visuomotor-only simulation experiments, and real-robot experiments replacing ArUco markers with marker-free perception, all included in the revised paper. (Section 5; Appendices B.1, C.5, and E.1).

Noise injection is intended to avoid reliance on privileged exact simulator-provided poses. In robotic practice, object poses can be estimated from RGB-D observations via pose estimation methods, which introduce estimation errors. The injected noise is intended to approximate this uncertainty, allowing us to evaluate execution with imperfect pose inputs rather than exact simulator ground truth.

RoboHarness’s ability to use a broader range of available information is itself a system-level advantage. RoboHarness integrates policies with different input requirements into a unified framework, assigning subtasks and invoking policies according to the available information. It can therefore use different information sources through compatible policies without architectural redesign or retraining. In contrast, purely visuomotor policies without pose inputs cannot directly use pose estimates and may require substantial redesign and training to incorporate them.

To further address this concern, we evaluate RoboHarness using only visuomotor policies on LIBERO-Plus, LIBERO-LoHo, and RMBench. RoboHarness still outperforms the individual policies on all three benchmarks.

MethodRobot StateLanguageLayoutBackgroundSensorCameraLightOverallΔ\DeltaOverall
RoboHarness (visuomotor policies only)90.096.886.595.289.778.897.690.10.0
π0.5\pi_{0.5}77.585.685.794.689.775.496.985.7-4.4
Faster-WAM58.390.981.760.784.676.594.278.3-11.8
OpenVLA-OFT21.781.068.791.078.655.692.767.9-22.2

Table 1: Success rates (%) on LIBERO-Plus.

On LIBERO-Plus, overall success reaches 90.1% without TAMP, outperforming all individual baselines. Instruction preprocessing and Memory Bridge support gains under language and robot-state perturbations, while dynamic policy assignment improves robustness to other perturbations.

MethodSR (%)PS (%)Δ\DeltaSRΔ\DeltaPS
RoboHarness (visuomotor policies only)54.883.20.00.0
π0.5\pi_{0.5}6.455.3-48.4-27.9
Faster-WAM0.424.9-54.4-58.3
OpenVLA-OFT0.020.0-54.8-63.2

Table 2: Success rate (SR) and progress score (PS) on LIBERO-LoHo.

On LIBERO-LoHo, RoboHarness improves long-horizon execution through task decomposition and policy coordination. The particularly pronounced gains in progress score show that, even when capability gaps prevent full task completion, RoboHarness can coordinate the available policies to achieve substantial partial progress.

MethodSR (%)Δ\DeltaSR
RoboHarness (visuomotor policies only)61.70.0
Mem-042.0-19.7
π0.5\pi_{0.5}10.4-51.3
X-VLA9.8-51.9
Diffusion Policy5.8-55.9

Table 3: Success rate (SR) on RMBench.

On RMBench, RoboHarness retains its advantage over baselines without TAMP or simulator-provided object poses, supporting the effectiveness of its memory-dependent execution beyond pose-based planning.

Our additional real-robot experiment shows that RoboHarness remains effective without accurate poses provided by ArUco markers. We unify the sensory input across TAMP and the visuomotor policies by replacing ArUco markers with a marker-free perception module, keeping all other settings unchanged. This module estimates object poses from the shared RealSense D435 RGB-D observations through height-map construction, object segmentation, and geometric matching against known block dimensions. An independent memory module tracks block identities over time. These poses are derived entirely from the shared visual input. RoboHarness achieves 73.3% success on the Bridge-building task with marker-free perception, compared with 86.7% using ArUco markers. Implementation details are provided in the appendix for your reference (Appendix B.1).

Our framework can coordinate varying numbers of heterogeneous policies without additional training. The policy library can contain two, four, or even more policies, depending on the availability of compatible implementations and checkpoints.

W2 + Q2

We address your concerns by providing an in-depth analysis of our ablation study (Fig. 4) and a comparison of pre-update success rates.

Orchestration provides substantial gains even with a frozen harness, while online adaptation further improves performance. In the ablation study from the original paper (Fig. 4), the No-Evolution setting disables both evolution updates and cross-episode memory accumulation, including across repeated trials of the same task. On LIBERO-Complex, this frozen variant achieves 53.4% SR, exceeding the strongest underlying policy (36.4%) by 17.0 percentage points. Enabling online adaptation further increases SR to 86.8%, an additional gain of 33.4 percentage points. These results support the complementary benefits of orchestration and online adaptation. We have expanded the ablation discussion in the Results section (lines 479–485).

Following your suggestion, we further examine how the gains of online updating vary across benchmarks. We inspected the evaluation logs and retained only trials that satisfied both conditions: (1) execution occurred before any harness or skill update, and (2) no memory accumulated from earlier evaluation trials was retrieved. We denote the success rate on these trials as pre-update SR and compare it with overall SR in Table 4.

BenchmarkStrongest baselinePre-update SROverallOverall −- Pre-update (pp)
LIBERO-LoHo64.885.096.0+11.0
LIBERO-Plus85.787.991.9+4.0
RMBench42.066.777.2+10.5
BEHAVIOR-1K23.340.043.3+3.3
Average (SR)54.069.977.1+7.2

Table 4: Pre-update and overall success rates (%) of RoboHarness, compared with the strongest baselines.

As shown in Table 4, pre-update RoboHarness outperforms the strongest baseline on all four benchmarks, demonstrating the effectiveness of orchestration even without online updates. Overall SR exceeds pre-update SR by 3.3–11.0 percentage points across benchmarks. The varying gains are consistent with differences in initial task coverage, with weaker coverage leaving more room for improvement through evolution. This descriptive comparison complements the controlled No-Evolution ablation, supporting the benefit of online adaptation. These results are included in (Appendix E.5) of the revised paper.

W3+Q3

We clarify how visual retrieval and downstream policy generalization address object displacement, while current-scene motion planning handles collision risks. We have revised (Sections 4.2; Appendix E.3) to provide further information.

Under object displacement, Memory Bridge guides the robot toward a handoff configuration similar to those in the incoming policy’s successful experience. Memory Bridge retrieves trajectories whose visual observations and task instructions most closely match the current scene and task. It then guides the robot toward a configuration within the neighborhood supported by these trajectories, bringing its starting configuration closer to those encountered during the incoming policy’s training or successful execution. After the handoff, the incoming VLA relies on its generalization capability and current visual observations to accommodate the remaining differences in object position. The progress estimator therefore does not need to specify the final grasp pose; its role is to establish a favorable starting configuration for the incoming policy.

Collision risks are addressed through the motion-cost term evaluated in the current scene. Our trajectory search objective combines the predicted progress score with a motion-cost penalty, fscore,t(s)−λmotionCmotion(st,s)f_{\mathrm{score},t}(\mathbf{s})-\lambda_{\mathrm{motion}}C_{\mathrm{motion}}(\mathbf{s}_t,\mathbf{s}). While the progress estimator uses robot proprioceptive states, a motion planner evaluates motion cost in the current scene. Transitions involving collisions, obstacle-induced detours, or greater planning effort incur higher costs, reducing their combined objective values even when their predicted progress is high. Thus, the Retrieved configurations guide a collision-avoiding neighborhood search rather than serve as fixed handoff targets.

The drawer-closing step in our original real-robot experiment provides a concrete example of handling scene mismatches. Visually similar retrieved frames contained configurations from partially closed-drawer states, whereas the current drawer was fully open. Directly planning to these configurations caused repeated planning failures because they collided with the open drawer. Memory Bridge penalizes these configurations through their high current-scene motion cost and searches for nearby alternatives within the memory-supported region, eventually finding a collision-free handoff configuration within the incoming policy’s familiar operating distribution.

W4 +Q4

We conduct additional experiments by switching the VLM backbones to address your concerns. The experiment and discussion are included in the revised paper (Appendix E.2).

RoboHarness supports open-weight coding agents, with backbone capability affecting the trade-off between task success and reasoning latency. We acknowledge that the additional reasoning required for orchestration increases execution latency. To assess sensitivity to the backbone and the potential for reducing this overhead, we evaluated different VLM backbones on the real-robot Build Bridge task, running 15 trials per backbone under the same protocol as in the paper. Only the backbone was changed.

BackboneOpen-weightSuccess (%)Latency / call (s)Overhead / trial (s)
GPT-6 AstraNo100.0 (15/15)8.1155.5
GPT-5.5 (paper)No86.7 (13/15)4.667.4
Qwen3.5-27BYes80.0 (12/15)4.347.3
Qwen3.5-4BYes40.0 (6/15)1.618.8

Table 5: Backbone comparison on the real-robot Bridge task. Qwen models are served locally with thinking disabled.

The 27B open-weight backbone retains strong performance, while the 4B model exposes limitations in spatial grounding. Qwen3.5-27B achieves one fewer successful trial than GPT-5.5, whereas Qwen3.5-4B frequently identifies the wrong block and issues an incorrect grasp call. GPT-6 Astra succeeds in all trials. These results suggest that reliable orchestration requires adequate spatial grounding and reasoning, which a 27B open-weight backbone can provide on this task.

Local open-weight deployment removes proprietary API dependence, while revealing a trade-off between latency and success. The 27B model offers modest latency savings, while the 4B model substantially reduces reasoning overhead at the expense of task success. Local deployment also avoids per-call API charges, although reasoning latency remains a limitation for dynamic or time-critical tasks. These findings are specific to the real-robot Build Bridge evaluation.

Thank you again for your thoughtful feedback and constructive suggestions. We hope our responses can address your concerns. Please let us know if any points require further clarification; we would be happy to discuss them further.

Reviewer_oKhdOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents RoboHarness for coordinating heterogeneous robot policies (including VLA, WAM etc.) The proposed model is comprised of several modular components, as illustrated in Fig. 1, including understanding skills, memory skills and evolution skills. These components are corresponding for reading policy capabilities, retrieving related episodes from previous records, and adapting policies respectively. In the outer control loop, it utilizes a coding agent for scheduing the computation. Experiments are conducted on five benchmarks and several real-world robot tasks.

Critical strengths

Overall I regard this as an interesting work on an important topic in recen trend of embodied robotics research. Below I would briefly state its critical strenths.

  1. The paper presents a promising idea - a single policy often is limited in its applicabiliyt and capability, while orchestration of multiple policies can potentially lead to enhanced performance. It develops a full solution with several modular components. Each of them is well designed and complementary to each other. The collection of all components seem a good solution for a variaty of long-horizon robotic manipulation tasks.

  2. The authors have identified several key practice challenges in above ideas. In particular, the gap between two consecutive policies is common, namely the robot may stay in a state at the end of previous policy that is not in the feasible region of the next policy. To this end, memory bridge is proposed, implemented by local score estimation and trajectory generation. This leads to a reasonable solution.

  3. The experiments are comprehensive and convincing, with 5 benchmarks and some real-robot tasks. The adopted policies are modern and cover the set of representative models.

Critical weaknesses
  1. My major concern is regarding the memory bridge. Its objective is a combination of two terms (scores and motion cost), modulated by a hyper-parameter. The idea is ad hoc and sensitivity of the hyper-parameter is not fully discussed. The key details of the solver for finding the trajectory is not presented in the main text. The part of local progress estimation is critical yet only superficially treated in the paper. It is hard to judge its effectiveness and generalization to challenging out-of-domain scenarios.

  2. The way of training the entire system, including the coding agent and all components, is not explained. The set of parameter to be optimized is not clearly specified. Regarding the components, a lot of designs are presented without explaining the underlying motivation, such as understing the policy (it is not clear to me the implication of input-quality assessment to a policy). Authors tend to define too many notations. Some of them are never used again after tis definition.

  3. There are some strong claim superficially made in the draft, yet no evidence is provided. For example, the ability of the coding agent and updating rules. Many key information is frequently deferred to the appendix. The paper should be self-contained for easy understanding.

  4. The experiments seem promising, particulary in Table 1. However, it lacks some in-depth analysis of the source of improvement. It is also unclear to me how the sub-tasks are defined on each benchmark. Some claims are made on empirical results and seem not general enough, such as those in Fig.3(a).

Questions for authors

See my detailed comment in section "critical weaknesses".

3 comments

Thank you for your positive assessment and constructive suggestions. We sincerely appreciate your recognition of RoboHarness’s practical contribution to heterogeneous policy coordination, Memory Bridge’s lightweight handoffs without joint fine-tuning, and our broad evaluation. Your feedback has helped clarify our design rationale, methodological details, and empirical evidence. We address your concerns below.

W1

Here, we explain Memory Bridge’s design rationale and solver, report additional hyperparameter sensitivity experiments, and analyze effectiveness and generalization.

Memory Bridge draws on established heuristic-guided search principles rather than an ad hoc design. Following heuristic-guided planning [1], local progress and motion cost serve as two heuristics that favor configurations suitable for the incoming policy while discouraging expensive transitions. These choices connect to optimal control, offline RL, and learning from demonstrations: motion cost and progress scoring are analogous to transition cost and terminal evaluation in optimal control [2]; the memory-supported region limits extrapolation, following support constraints in offline RL [3]; and temporal ranking provides relative progress supervision, consistent with learning rewards from temporal comparisons [4].

Trajectory solver. The solver iteratively samples configurations within the memory-supported confidence region. The local estimator predicts progress; OMPL evaluates reachability and motion cost. Candidates without collision-free connecting trajectories are penalized through the motion cost term. Candidates are ranked by the combined objective, and the highest-ranked becomes the next sampling center. Search stops at the score threshold, subject to a 10-second limit; the motion planner generates a trajectory to the selected target. The default threshold is 0.9 and λmotion=0.2\lambda_{\mathrm{motion}}=0.2, both agent-adjustable. If no acceptable candidate is found, the threshold is lowered in steps of 0.02. We moved key solver details from the appendix to the main text (Section 4.2.2; Appendix C.2).

Hyperparameter sensitivity is limited over the tested settings. Across five benchmarks, the agent modified λmotion\lambda_{\mathrm{motion}} in only 4.8% of evaluation instances. We also evaluated fixed weights on the ablation-study dataset:

SettingSR (%)PS (%)
Default 0.2, agent-adjustable86.894.2
λmotion=0.2\lambda_{\mathrm{motion}}=0.2, fixed84.493.0
λmotion=0.5\lambda_{\mathrm{motion}}=0.5, fixed87.293.8

These results suggest precise weight tuning is unnecessary within the tested range. λmotion\lambda_{\mathrm{motion}} controls the trade-off between progress and planning effort; the score threshold determines when a candidate is acceptable for stopping search. The sensitivity test is included in the revised draft (Appendix C.2, Table 4).

Local Progress Estimation. For each anchor retrieved from successful trajectories using the current subtask and observation, we collect preceding and subsequent states within a predefined temporal horizon, stopping at subtask or gripper-state changes. Each sample pairs a normalized robot-state vector with its signed temporal offset from the anchor. These offsets provide pairwise supervision for a linear ranking SVM, fscore,t(s)=wt⊤sf_{\mathrm{score},t}(\mathbf{s})=\mathbf{w}_t^{\top}\mathbf{s}. Training balances weight regularization against ordering violations, favoring states with larger temporal offsets. Fitted separately for each bridge, the estimator measures relative local progress and discourages revisiting earlier stages requiring repeated motions. We clarified this in the main text (Section 4.2.2; Appendix C.2).

We evaluate Memory Bridge’s effectiveness and generalization through longer unseen tasks, out-of-distribution perturbations, and a controlled ablation.

First, LIBERO-LoHo tests unseen task sequences approximately four times longer than training tasks, requiring multi-round coordination and policy handoffs. RoboHarness achieves 96.0% success (Figure 3).

Second, LIBERO-Plus tests out-of-distribution conditions across seven perturbation dimensions. Robot-state perturbations are particularly relevant to Memory Bridge: they place the robot in unfamiliar configurations from which suitable policy-execution states must be recovered. RoboHarness achieves 90.4% success in this category (Table 1).

Third, the LIBERO-Complex ablation isolates Memory Bridge’s contribution by removing it while retaining other components. Success drops from 86.8% to 60.8% (Figure 4). These results support system-level generalization and demonstrate Memory Bridge’s contribution to successful execution.

We expanded the results analysis to clarify effectiveness and generalization (Section 6).

W2

To clarify the system design, we detail the system setup, identify the tunable parameters, explain the motivations behind the skills, and simplify the notation. If there are other design choices you would like us to explain further, please let us know, and we will add the corresponding explanations.

RoboHarness requires no joint training of the coding agent and robot policies. Our goal is to coordinate independently developed policies through a frozen pretrained VLM-based coding agent. The underlying policies are pretrained or designed separately. Since our method focuses on using these policies and does not include their training, we document checkpoint sources and independent training procedures in Appendix C.5. During orchestration, the network weights of the learned policies, coding agent, and encoders remain frozen. The parameters fitted or tuned within RoboHarness are the local progress estimator’s ranking-SVM weights, as described in our response to W1, and a limited set of exposed tool parameters, which we specify below. We have revised (lines 150–156; Appendix C.1) to clarify.

The motivation behind understanding skill design is to assess the conditions required for reliable policy execution. Uncertainty assessment is motivated by the sensitivity of geometry-dependent policies to inaccurate object poses (Are the pose estimates reliable enough for geometry-based execution?). Visual- and semantic-context assessments are motivated by the need to match the current scene and requested subtask to each policy’s experience and capabilities (Is this scene familiar to the policy, and does the subtask match its capabilities?). Input-quality assessment accounts for the dependence of vision-based policies on clear, informative images, as blur or poor exposure can impair execution even on familiar tasks (Are the images clear enough for reliable visual control?). State-policy compatibility assessment addresses the risk that one policy’s terminal configuration is unsuitable for the next (Can the next policy start from this configuration, or is a Memory Bridge needed?). We have clarified these motivations in the main text (lines 160–175)

Tunable parameters Visual-context assessment combines two visual encoders and therefore exposes a mixing weight for tuning. State-policy compatibility assessment uses the same ranking SVM as Memory Bridge and therefore requires local fitting on samples from retrieved trajectories. Uncertainty, semantic-context, and input-quality assessments use fixed statistical computations or frozen-encoder similarity calculations and therefore have no parameters to tune in our implementation. We have added a table listing the parameters fitted or tuned (Appendix C.1, Table 3).

Notation. We have removed unnecessary definitions and simplified the notation, as shown in (Section 3).

W3

We provide further support for our claims about coding-agent updates through expanded experimental analysis, methodological references, and clearer descriptions of the update procedures and the scope of the supporting evidence.

Coding-agent refinement of interfaces and routing. CodeAct [5] and SWE-agent [6] demonstrate that coding agents can use execution feedback to diagnose errors and revise code. RoboHarness applies this established capability to robotic policy orchestration, using execution traces and runtime errors to refine policy-interface wrappers and routing routines. We have added these references to clarify the methodological basis of harness refinement (lines 308–312).

Improved execution through online updates. In the ablation study presented in the paper, removing evolution reduces success from 86.8% to 53.4% (Figure 4), demonstrating the contribution of online updates to successful execution. We have expanded the Results discussion to explain this finding and more clearly connect the evidence to our claims (lines 479–485).

Updating Rules. We have included the essential update procedures in the main text to make the presentation self-contained, with further implementation details in Appendix C.4 (Section 4.4; Appendix C.4). We summarize them below:

  1. Code refinement is restricted to policy-interface wrappers and routing routines, including input/output handling, policy-selection conditions, retry rules, and fallback logic. The predefined information-flow framework remains fixed.

  2. Hyperparameter updates are restricted to exposed skill/tool parameters and follow a directional search over discrete candidate values.

Both types of updates are evaluated through subsequent execution, with changes and outcomes recorded in the execution log to inform further updates.

If additional claims or technical details require further clarification, we would appreciate your feedback so that we can address them specifically.

W4

In response, we have expanded the discussion of the sources of improvement, clarified subtask definitions, and refined the interpretation of our results (Section 6). The added analysis is summarized below.

We have revised and expanded the Results discussion to clarify the sources of improvement across benchmarks. On LIBERO-Plus and LIBERO-LoHo, capability-aware decomposition and routing exploit complementary policy coverage to handle perturbations and subtasks that a single policy cannot reliably address. On RMBench, memory management and retrieval separate historical-context handling from low-level execution, while complementary policies improve subtask reliability. Memory Bridge supports compatible handoffs across these settings, particularly under robot-state perturbations in LIBERO-Plus and during navigation-to-manipulation transitions in BEHAVIOR-1K. Finally, the LIBERO-Complex ablations support the complementary contributions of contextual assessment, online updates, and memory-guided handoffs.

Subtask definition. Across benchmarks, subtasks correspond to action primitives such as picking, placing, and moving, following the standard task-level action abstraction in robotic task planning [7]. For benchmarks that provide ground-truth substeps, such as LIBERO-LoHo and BEHAVIOR-1K, we follow their definitions; otherwise, we follow conventional robot task-planning decomposition. The coding agent decomposes the requested task into these executable units and assigns each to an appropriate policy based on its capabilities.

Clarifying the interpretation of Fig. 3(a). We have revised the discussion to connect the empirical evidence more explicitly to our conclusion and ground our claims in the observed results. Across seven perturbation categories, VLA invocation generally increases with its standalone success rate, indicating that RoboHarness identifies context-dependent policy strengths and favors the more reliable policy under the current conditions. Policy cards provide initial capability information, while accumulated rollout experience refines these assignments. This ability to exploit complementary policy strengths helps explain the observed robustness gains. Sensor noise remains an exception because blurred observations impair the assessment of policy suitability. (lines 398–407)

References

[1] Hart et al. (1968). A Formal Basis for the Heuristic Determination of Minimum Cost Paths. IEEE Transactions on Systems Science and Cybernetics.

[2] Rawlings et al. (2017). Model Predictive Control: Theory, Computation, and Design. Nob Hill Publishing.

[3] Fujimoto et al. (2019). Off-Policy Deep Reinforcement Learning without Exploration. ICML.

[4] Burke et al. (2023). Learning Rewards from Exploratory Demonstrations Using Probabilistic Temporal Ranking. Autonomous Robots.

[5] Wang et al. (2024). Executable Code Actions Elicit Better LLM Agents. ICML.

[6] Yang et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS.

[7] Garrett et al. (2021). Integrated Task and Motion Planning. Annual Review of Control, Robotics, and Autonomous Systems.

Thank you again for your constructive feedback and assessment. Please let us know whether we have addressed your concerns or if any further clarification would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.