acceptodds
Under review as a conference paper at ICLR 2027

EVA: A Memory-Augmented Harness for Long-Horizon Embodied Tasks

11 views

Abstract

Recent advances in Vision-Language-Action (VLA) models have improved the generalization of embodied foundation models, yet their performance remains limited on tasks requiring long-horizon reasoning, persistent memory, and robust execution under environmental noise. Moreover, consolidating diverse capabilities into a single model is costly and difficult to scale. Inspired by general-purpose agent harnesses, recent works have explored agentic frameworks for embodied tasks, but these approaches largely rely on directly adapting general-purpose harnesses, leaving the design of a dedicated, efficient, and general-purpose embodied harness largely unexplored. Therefore, we introduce **EVA**, a general and extensible embodied harness that provides a lightweight set of essential components, integrating perception, memory, memory-grounded planning, execution, monitoring, and verification. At each step, EVA generates an executable subtask and selects an appropriate executor, monitors its execution online, verifies the outcome, and updates memory to enable replanning or recovery. This decoupled design enables existing embodied models to achieve robust, persistent, and recoverable long-horizon behaviors without modifying their parameters or requiring test-time adaptation. We further theoretically characterize the benefits of the harness-based approach and conduct extensive experiments across distribution shifts, long-horizon reasoning, and embodied-memory benchmarks. EVA achieves an average improvement of **37.8%** over strong baselines while reducing token overhead by **10.9** on VLABench. Real-world deployment on a humanoid robot further demonstrates its effectiveness. Overall, these results show that \ours can substantially extend the capabilities and generalization of existing embodied models. Code is available at [https://anonymous.4open.science/r/eva-0](https://anonymous.4open.science/r/eva-0).

Primary area:
applications to robotics, autonomy, planning
Keywords:
Harness, Memory, Robotics, Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pBxXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops a harness built around different types of memory to improve manipulation performance. It also augments the VLA with additional capabilities supported by foundation models, ultimately achieving substantial improvements on several long-horizon tasks.

Critical strengths
  1. The paper achieves solid performance improvements and includes supporting real-world robot experiments.
  2. The overall narrative is clear, making the key aspects of the design easy to follow.
Critical weaknesses
  1. The paper lacks visual examples, including visualizations of execution processes in the experiments and diagrams that facilitate understanding and comparison.
  2. The paper does not include comparisons on a broader range of commonly used benchmarks.
  3. The harness designs are fairly common. More comprehensive comparisons with similar designs in other agentic approaches are needed to understand where the final performance gains come from.
Questions for authors
  1. The designs of the individual harness components are fairly common. How does the proposed approach differ from similar work in its design, and where do the final performance gains come from?
  2. Should results on more commonly used benchmarks be included? Even for a method designed for long-horizon tasks, performance on general-purpose tasks remains important.
  3. What additional overhead does the harness introduce compared with end-to-end models? This warrants a dedicated analysis. Whether the harness yields different gains when paired with different underlying models also requires thorough examination.
2 comments
AuthorsOpenReview ↗

Response to Reviewer pBxX (Part 1).

We would like to express our sincere gratitude for your valuable feedback and comments and we address your concerns as follows.

W1: The paper lacks execution visualizations and diagrams that facilitate understanding and comparison.

Thank you for this insightful suggestion. We will provide a dedicated demo page https://eva-anonymous.github.io/ featuring visual demonstrations of the effectiveness of each component, simulation performance comparisons between EVA and other baselines, and real-world robot experiments. These demonstrations will offer a more intuitive understanding of EVA's component-level contributions, comparative performance, and real-world applicability.

W2 & Q2: Should results on more commonly used benchmarks be included? Even for a method designed for long-horizon tasks, performance on general-purpose tasks remains important

We agree that evaluating general-purpose performance is important. In fact, our existing evaluations on VLABench, RoboMME, and LIBERO-LoHo already cover a range of general-purpose manipulation tasks like picking and placing objects, pressing buttons, opening and closing drawers, inserting flowers, and adding condiment. For instance, VLABench includes diverse general-purpose manipulation tasks and evaluates robustness beyond individual long-horizon sequences. The results on these benchmarks already demonstrate EVA's strong performance, indicating its capability to handle general-purpose manipulation tasks effectively.

To further address the reviewer's concern regarding performance on general-purpose benchmarks, we additionally evaluate EVA on LIBERO-PRO[1] benchmark, covering four suites with 10 tasks each under five perturbation types. We instantiate EVA with π0\pi_0 VLA policy and compare EVA with raw π0.5\pi_{0.5} policy to further evaluate its improvement on general-purpose tasks. We use initial state 0-1 per task–perturbation pair (seed 0), yielding 400 rollouts per method. The results are presented below.

Table 6. Results on LIBERO-Pro (success rate, %).

MethodSwapObjectLanguageTaskEnvironmentAverage
π0\pi_00.0055.0030.0017.508.7522.25
EVA (π0\pi_0)0.0058.7552.5022.506.2528.00

The results show that EVA with π0\pi_0 outperforms the raw π0\pi_0 baseline across three perturbation types object, language and task on LIBERO-PRO, achieving an relative success rate improvement of 25.8%, confirming that our framework not only excels in long-horizon tasks but also enhances general-purpose manipulation capabilities in different benchmarks.

W3 & Q1: How does EVA differ from similar harness designs, and where do the gains come from?

Thanks for giving the chance for clarification. The novelty of EVA is not an isolated planner, memory module, or verifier. It is an embodied execution protocol that gives each subtask explicit transition semantics. For a subtask zz, EVA specifies not only how it is executed, but also when execution should stop, how completion is verified, how task and object state are updated, and how the system should continue or recover after failure. This makes “the rollout has stopped,” “the subtask has succeeded,” and “the robot is ready for the next subtask” three distinct states, rather than treating them as one event.

This perspective distinguishes EVA from the compared systems. RoboClaw and ABot-Claw primarily organize high-level planning and persistent memory, but do not formulate this explicit execution–termination–verification–recovery transition around every subtask. Harness VLA and Volo use agentic memory to steer a frozen VLA, but does not provide EVA’s separation between high-frequency online termination and post-execution semantic verification, nor its object-centric state update in the execution loop.

Concretely, EVA implements this protocol through (i) task memory for progress and verified outcomes, (ii) object memory for persistent identities and changing spatial states, (iii) event-driven monitoring that terminates an execution when a physical condition is met, (iv) semantic verification after termination, and (v) state-aware replanning or physically valid recovery. These mechanisms are coupled: verification updates memory, memory determines the next subtask and its termination conditions, and recovery changes the robot state without resetting the scene. The contribution is therefore the system-level abstraction and its concrete realization for embodied interaction, not the claim that each component is individually new. Ablation studies in Appendix D further demonstrates the effectiveness of these design choices from an empirical perspective.

AuthorsOpenReview ↗

Response to Reviewer pBxX (Part 2).

Q3(a): What overhead does the harness introduce relative to an end-to-end model?

Thank you for this question. As analyzed in Appendix D (Deployment Timeline Analysis, Figure 5), EVA introduces additional latency primarily at subtask transitions, mainly from high-level planning and outcome verification, while checkpoint recovery is comparatively short. Importantly, EVA mitigates this overhead through a lightweight monitor that triggers semantic verification only when execution termination is detected, rather than at every control step. Furthermore, perception is processed asynchronously to overlap with other stages where possible. Thus, EVA concentrates additional reasoning overhead at critical decision boundaries while minimizing unnecessary synchronization, enabling more reliable long-horizon execution without repeatedly invoking expensive high-level models.

Q3(b): Does the benefit depend on the underlying models?

We investigate the dependence of EVA's benefits on the underlying models from two perspectives: (1) low-level VLA policy and (2) high-level planning and verification models.

For low-level VLA policy, as shown in Table 3, on VLABench, the success rate improves from 29.4% to 43.2% with π0.5\pi_{0.5} and from 65.8% to 73.2% with πXR1\pi_{\mathrm{XR1}}. These results demonstrate that EVA is applicable to different low-level VLA policies with various improvement.

As for high-level planning and verifier, We further conduct additional experiments on LIBERO-LoHo by varying the underlying models used for planning and verification. Specifically, we compare three model configurations: Qwen3.8-27B (a smaller open-source model), DeepSeek-V4.1-Flash (a larger open-source model), and GPT-6-Astra (a frontier closed-source model). The results are reported in the table below.

Table 7. Performance of EVA with different LLM backbones on LIBERO-LoHo. We report Q-Score (QS, %) and success rate (SR, %). The best and second-best results in each column are highlighted in bold and underline‾\underline{underline}, respectively.

LLM BackboneTask 1 / QS ↑\uparrowTask 1 / SR ↑\uparrowTask 2 / QS ↑\uparrowTask 2 / SR ↑\uparrowTask 3 / QS ↑\uparrowTask 3 / SR ↑\uparrowTask 4 / QS ↑\uparrowTask 4 / SR ↑\uparrowTask 5 / QS ↑\uparrowTask 5 / SR ↑\uparrowAverage / QS ↑\uparrowAverage / SR ↑\uparrow
Qwen3.8-27B88.365.0‾\underline{65.0}78.355.081.760.0‾\underline{60.0}96.790.080.035.085.061.0‾\underline{61.0}
DeepSeek-V4.1-Flash93.3‾\underline{93.3}90.095.085.091.775.095.0‾\underline{95.0}90.083.8‾\underline{83.8}45.0‾\underline{45.0}91.777.0
GPT-6-Astra95.090.090.0‾\underline{90.0}75.0‾\underline{75.0}88.3‾\underline{88.3}75.090.080.0‾\underline{80.0}85.065.089.7‾\underline{89.7}77.0

The results show that the Qwen-based configuration achieves a 61% success rate, while the DeepSeek- and Astra-based configurations achieve 77%. This demonstrates that EVA maintains good performance even when using a relatively small model such as Qwen3.8-27B. Moreover, we observe no significant performance difference between the two stronger models and no single model consistently dominates across all tasks.

To better understand the performance, we further conduct trajectory analysis. Our analysis suggests that the lower success rate of Qwen3.8-27B is primarily associated with less reliable completion-state verification and recovery planning. In particular, these models are more likely to incorrectly assess whether a task has been completed or fail to generate effective recovery plans when execution deviates from expectations. This suggests that improving VLMs' capabilities in completion-state verification and recovery planning could further enhance the overall system performance.

Overall, these results suggest that EVA's benefits are not limited to a particular underlying model. While the absolute performance may vary with model capability, EVA remains effective across different low-level policies and high-level planning and verification models.

Reference:

[1] Zhou et al., “LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization,” arXiv, 2025.

Reviewer_KFFHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EVA is a memory-augmented execution harness for long-horizon embodied tasks. It maintains task and object memory, uses this memory to generate the next subtask and select an executor, and coordinates execution through lightweight online monitoring and event-triggered semantic verification. Verification outcomes update memory and guide subsequent planning or recovery. The framework can wrap existing policies without updating their parameters during deployment. The paper evaluates EVA on VLABench, RoboMME, and LIBERO-LoHo, reports gains over the underlying policies and other agentic systems, and provides component ablations and an illustrative humanoid deployment. A simplified probabilistic analysis characterizes verification and retry behavior. The main contribution is an integrated execution architecture for extending existing embodied policies.

Critical strengths
  1. The complete harness provides useful gains across two underlying policies.

On VLABench, EVA improves average success from 29.4% to 43.2% with π0.5 and from 65.8% to 73.2% with πXR1 (Table 2). These within-policy comparisons provide more direct evidence for the practical value of execution management than comparisons between systems using different policies. The results indicate that the complete harness can improve task completion without updating the deployed policy parameters. This is a substantive practical strength, although it does not by itself establish the originality of the architecture or isolate a particular new mechanism.

  1. Separating termination monitoring from semantic verification is practically motivated and supported by ablations.

The design recognizes that deciding when to stop a policy rollout and deciding whether the subtask succeeded are different operations (§3.4). Lightweight monitoring can stop undesired continued motion without invoking a semantic VLM at every control step. On the 50-rollout LIBERO-LoHo ablation set, success falls from 78% to 40% without the monitor, to 70% without the verifier, and to 32% without recovery (Table 7). These results support the practical importance of execution management. The monitor ablation retains only timeout termination, so it establishes the value of monitoring over that fallback rather than the superiority of a particular monitoring algorithm.

Critical weaknesses
  1. Limited methodological novelty beyond integrating established agent components.

The core architecture combines familiar elements: object and task memory, language-model planning, policy or primitive selection, outcome checking, and recovery. The paper’s own related-work discussion and Table 1 show substantial overlap with existing embodied agents. Earlier work such as Inner Monologue already uses success detection and environmental feedback for closed-loop language-model planning. EVA’s clearest distinction is the particular separation and coordination of monitoring and verification, together with its memory organization. This is a useful architectural refinement, but the paper offers limited new mechanisms for memory updating, executor selection, or recovery beyond assembling these functions.

The theoretical analysis does not substantially strengthen the originality: it studies a stationary retry process with fixed execution and verification probabilities, rather than the distinctive memory or scheduling mechanisms of EVA. The resulting contribution is primarily an integration of established functions, with limited conceptual advancement. My concern is the modest intellectual increment of the proposed design, not a requirement that every systems paper introduce a new learning algorithm.

  1. The systems contribution provides limited new insight beyond confirming the utility of standard execution-management functions.

A systems paper can make a substantial contribution through a useful new abstraction, a non-obvious design trade-off, or a generalizable empirical finding. Here, the main lesson is that adding memory, timely termination, verification, and recovery improves execution. The within-policy gains are valuable, but the component-removal studies largely confirm the importance of these familiar functions. In particular, the monitor ablation replaces all termination rules with timeout alone; its large performance drop demonstrates the need for timely stopping, rather than a distinctive advantage of EVA’s monitor–verifier coordination. Similarly, the recovery ablation shows that allowing recovery helps, but does not establish a new recovery principle. The findings offer limited guidance on when or why EVA’s particular organization is preferable to other reasonable implementations of these functions.

Recovery, which has the largest ablation effect, is also described mainly as returning to a stable checkpoint via a joint-space trajectory (§3.3; Appendix C). This does not explain how displaced objects are handled or what state is actually restored, limiting the transferable lesson from the strongest component result. Overall, the paper presents a useful working integration, but the resulting design insight and research significance are not substantial enough to offset the limited methodological novelty. This concern would remain even if implementation details were clarified.

Questions for authors
  1. What is the principal new mechanism beyond combining existing memory, planning, verification, and recovery components? In particular, which implemented operation distinguishes the monitor–verifier coordination from conventional termination detection followed by outcome checking, and which existing result supports that distinction?
  2. What generalizable design insight follows from the reported results beyond the usefulness of adding memory, stopping rules, verification, and recovery? Please identify an existing comparison or failure analysis that explains why EVA’s particular organization matters, rather than only showing that removing a function reduces performance.
  3. What exactly does a recovery checkpoint contain, and what is restored? Please use an existing rollout involving a moved object to explain the recovery actions, whether any simulator-state restoration or manual intervention occurs, and what recovery capability is specific to EVA.

If these three questions can be reasonably explained, I am willing to increase my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer KFFH (Part 1).

We greatly appreciate the time and effort you have dedicated to reviewing our paper. We address your concerns as follows.

W1: The method mainly integrates established agent components, and the stationary retry theory does not establish the originality of its memory or scheduling mechanisms.

Thank you for the constructive comments. We clarify our contributions from two aspects.

(1) Theoretical analysis. We agree that the analysis in Section 4 is indeed targeted at the benefits of the harness-based approach. As stated in the main text, it provides a supporting analytical rationale for verification-guided retries under uncertain execution outcomes and explains the necessity of the harness. For memory and planning, theoretically characterizing their benefits is challenging, as their effects are intertwined with planning, execution, and evolving environmental states. We will further clarify the scope of our analysis.

(2) Methodological novelty. We acknowledge that some of EVA's components share similar functional motivations with existing work, and do not claim individual novelty in memory, planning, executor selection, or recovery. However, similar components do not imply the same architecture or execution workflow. EVA organizes them into a closed-loop subtask execution process: memory-grounded planning generates a subtask and selects its executor; an event-driven Monitor determines when execution should terminate; a semantic Verifier assesses whether the subtask has succeeded; and the outcome updates task and object memory to guide subsequent planning or recovery. This physical-state-aware subtask transition is central to EVA's contribution. We further elaborate on the underlying mechanisms and provide illustrative rollout comparisons in our response to W2, and hope these details further clarify how EVA differs from existing approaches.

Overall, EVA's contribution lies in the coordinated execution workflow and its practical realization, rather than in individual components. We will revise the manuscript to better illustrate this system-level contribution and its distinction from related embodied harnesses.

W2: The ablations mainly show that familiar functions help, not why EVA's particular organization is preferable. What transferable insight follows?

Thank you for giving us the opportunity to illustrate the novelty. We would like to clarify that EVA's contribution does not lie in claiming novelty for each individual function, such as memory, monitoring, verification, or recovery. Rather, it lies in designing and implementing an integrated, extensible, and practically deployable execution-management framework for long-horizon robotic manipulation.

1. System-level design for physical-state-aware subtask transitions. EVA addresses a fundamental challenge in long-horizon manipulation: successful completion of one subtask does not necessarily guarantee either a timely transition or a suitable physical configuration for the next. EVA explicitly manages these transitions through complementary mechanisms. Event-triggered monitoring determines when the current execution should terminate, while checkpoint-based recovery provides a means of adjusting the robot's configuration before subsequent execution when necessary. Meanwhile, memory and multimodal perception provide persistent, physically grounded information to support planning, monitoring, and verification.

The importance of this organization is illustrated by rollout comparisons. (1) In LIBERO-LoHo task 0 with initial state 4, both full EVA and w/o-monitor register the butter grasp at step 59. EVA stops at step 70 and completes the task at step 340. Without the monitor, execution continues and closes the drawer at step 177 while the pudding remains outside. This demonstrates that timely termination preserves the physical boundary for outcome verification and subsequent subtask decisions, rather than merely reducing execution latency. (2) In task 4 with initial state 5, full EVA performs a 29-step checkpoint return before successfully grasping the pudding and completing the task at step 1053. Without recovery, the robot makes seven unsuccessful grasp attempts and exhausts the 3000-step budget. This illustrates that successful completion of one subtask does not necessarily produce a suitable starting configuration for the next. Together, these cases demonstrate why explicit management of physical-state-aware subtask transitions matters. Detailed demonstrations can be found on https://eva-anonymous.github.io/.

AuthorsOpenReview ↗

Response to Reviewer KFFH (Part 2).

2. Efficient and extensible component implementation. Beyond the overall architecture, EVA emphasizes efficient implementations of its components. For example, memory provides persistent task-relevant information, while the monitor uses executable DSL-based rules to perform high-frequency physical-condition checks without requiring online VLM reasoning at every step. Similarly, recovery uses Ruckig-generated jerk-limited trajectories to return the robot toward a saved configuration through ordinary environment actions, without privileged simulator-state restoration or environment reset.

We do not claim that each underlying function is independently unprecedented. Rather, the contribution lies in how these complementary capabilities are instantiated and organized to address the coupled requirements of long-horizon execution. The components are modular and extensible, allowing individual implementations to be improved or replaced without changing the overall execution-management principle. Their value should therefore be assessed not only individually, but also in terms of their roles and interactions within the complete system.

3. Practical applicability and real-world validation. Importantly, EVA is not merely a conceptual integration or a simulation-only framework, unlike many other existing works. Its components are implemented as an executable system and validated in both simulation and real-world robotic experiments. The efficient implementation of memory, monitoring, and other execution-management functions enables the framework to operate under practical runtime constraints. This demonstrates that EVA's design is not only conceptually motivated but also practically realizable and applicable to physical robotic systems.

We will revise the manuscript to clarify the architectural rationale, component interactions, implementation efficiency, and recovery procedure, and to incorporate the above rollout analyses. We believe these clarifications better establish EVA's contribution as an integrated, extensible, and practically validated framework for long-horizon robotic manipulation.

Q1: Which implemented operation distinguishes monitor--verifier coordination from conventional termination detection followed by outcome checking, and which result supports that distinction?

Thank you for providing us with the opportunity to clarify this point. Conventional termination methods, such as monitor–verifier coordination, typically rely on periodic polling to trigger verification at fixed intervals, or integrate termination and verification into a single, computationally expensive VLM-based verifier. Both approaches result in delayed termination, which causes the executor to execute incorrect subtasks during the gap between the actual completion of a subtask and the agent's termination. Additionally, frequent but redundant verification significantly increase computational overhead.

In contrast, EVA decouples high-frequency monitoring from low-frequency verification through an event-triggered coordination mechanism. For each subtask, the planner selects and parameterizes predefined termination rule templates or program custom monitoring rules using the monitor's domain-specific language (DSL), in which each monitoring rule is an executable evaluate(ctx) function. The runtime repeatedly invokes this function during execution, providing a structured context containing the latest SAM3-derived visual signals, robot proprioceptive signals, rule parameters, and persistent state for tracking temporal conditions. When a rule triggers, the monitor issues a termination signal to stop execution without online VLM reasoning. This mechanism supports flexible termination detection across a wide variety of subtasks through customizable rules. Moreover, by relying on lightweight function evaluation rather than expensive VLM inference, it achieves low-latency detection. On LIBERO-LoHo, removing the Monitor reduces success rate from 78.0% to 40.0%, supporting the importance of condition-triggered termination.

AuthorsOpenReview ↗

Response to Reviewer KFFH (Part 3).

Q2: What generalizable design insight follows from the reported results beyond the usefulness of adding modules? Identify an existing comparison or failure analysis that explains why EVA’s particular organization matters.

A central design insight is that a long-horizon harness must keep its decisions aligned with the evolving physical world, both through the information available to its components and through the subtask transitions it manages.

1. Provide a persistent, physically grounded information basis across the harness. EVA combines VLM-based semantic registration and SAM3-based binding and tracking with proprioception and task history. These complementary sources support different decisions: object identities and states ground planning, low-latency physical signals guide monitoring, and visual and physical evidence support outcome verification. Perception and memory thus inform the execution loop rather than serving only as context for the planner.

2. Manage subtask transitions explicitly. Semantic decomposition alone does not guarantee physical composability: the harness must control both when the current rollout ends and how the next begins. Two comparative analyses of existing ablation trajectories illustrate these requirements:

Comparative analysis A—timely termination keeps executor behavior aligned with the current planned subtask. In LIBERO-LoHo task 0, initial state 4, full EVA and w/o monitor issue the same initial VLA instruction, and trajectory diagnostics register the butter grasp at step 59 in both runs. EVA stops at step 70 and completes the task at step 340. Without the monitor, execution continues to step 502, closing the drawer at step 177 while the pudding remains outside. The verifier consequently assesses the original grasp after execution has already crossed that stage. This comparison shows that stopping is not merely a latency optimization: it preserves the physical boundary at which the harness should verify the outcome and decide what happens next, which keeps executor behavior aligned with the current planned subtask.

Comparative analysis B—successful completion of one subtask does not guarantee a suitable starting configuration for the next. In task 4, initial state 5, both variants place the butter before attempting to grasp the same pudding. Full EVA performs a 29-step physical checkpoint return, successfully grasps the pudding, and completes the task at step 1053. Without recovery, the robot proceeds directly from the placement configuration; seven pudding-grasp attempts are judged unsuccessful, and the final attempt exhausts the 3000-step budget. The footage shows repeated unproductive motions near the cabinet while the pudding remains on the table. This comparison motivates choosing the next starting configuration according to the upcoming subtask—not merely the previous success/failure label: preserve the held-object state for grasp-to-place transitions, but reposition when needed before a new grasp.

Detailed visual demonstrations can be found on https://eva-anonymous.github.io/.

Q3: What exactly does a recovery checkpoint contain, and what is restored? Please use an existing rollout involving a moved object to explain the recovery actions, whether any simulator-state restoration or manual intervention occurs, and what recovery capability is specific to EVA.

Thank you for raising this point. A recovery checkpoint stores the robot's joint configuration and gripper state, rather than the privileged simulator state. For example, in an existing rollout where the robot has displaced an object during manipulation, if it reaches a configuration poorly represented in the VLA's training data, EVA uses the saved checkpoint as a reference for recovery. Specifically, Ruckig [1] generates a jerk-limited joint-space trajectory, subject to velocity, acceleration, and jerk constraints, to guide the arm back toward the saved configuration with the gripper open. This recovery trajectory is executed through ordinary environment actions and consumes the episode's step budget.

Importantly, no simulator-state restoration or manual intervention is involved: the displaced object is not reset, and the environment continues evolving under normal physics. Only the robot's configuration is recovered, while the consequences of previous interactions are preserved. EVA thus enables checkpoint-based robot recovery within an ongoing manipulation episode without resetting the environment.

Reference:

[1] Berscheid and Kröger, “Jerk-limited Real-time Trajectory Generation with Arbitrary Target States,” RSS, 2021.

Reviewer_dccgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EVA, a training-free harness around frozen VLA/WAM policies. It combines object and task memory (VLM registration plus SAM3 tracking), a closed-loop planner that outputs one subtask and an executor at a time, a fast monitor that stops execution, a slower VLM verifier, and recovery to a stored checkpoint. A short probabilistic analysis shows that retrying with a better-than-chance verifier raises success. EVA improves average SR on VLABench (+7.4 with πXR1, +13.8 with π0.5), reaches 61.25 on RoboMME vs. 39.38 for Harness VLA, and reaches 79.0% SR on LIBERO-LoHo vs. 64.8% for the best baseline. A humanoid robot demo is shown qualitatively.

Critical strengths
  1. Consistent gains without retraining. The same harness improves two VLA backbones on VLABench and lifts the base policy on LIBERO-LoHo from 3% to 79% SR, showing that orchestration and verification can extend the horizon of existing VLAs.
  2. Informative ablations. Removing the monitor (−38 SR) or recovery (−46 SR) hurts far more than removing the verifier (−8), and removing memory drops RoboMME from 61.25 to 20.63. This shows where the value of such a harness comes from.
Critical weaknesses
  1. The headline claims compare across backbones. The abstract's "37.8% improvement" and "10.9× fewer tokens" compare EVA on πXR1 (73.2 SR) with Harness VLA (35.4 SR). EVA on π0.5 reaches only 43.2, so the like-for-like gain may be much smaller if Harness VLA uses π0.5.
  2. Recovery may use privileged simulator resets. Sec. 3.3 describes recovery as a joint-space motion of the robot, but App. C says it "restores the robot and environment to a stable state". Recovery is the largest ablation effect, so resetting object state in simulation would make the LIBERO-LoHo comparison unfair.
  3. Evaluation is small and mixed. EVA uses 10 episodes per task on VLABench and RoboMME with a single seed, while copied VLA baselines use the official 50. Latency and tokens are averaged over successful episodes only.
  4. The real-robot result is anecdotal. It is a single qualitative example with no success rate or trial count, yet the abstract presents it as evidence of effectiveness.
  5. Limited novelty. The components overlap heavily with RoboClaw, ABot-Claw, Harness VLA and VoLo. The theory (Props. 4.1 to 4.4) is a generic retry model: every attempt succeeds with the same probability p and is judged by a symmetric verifier with accuracy q, which gives a geometric series. It does not involve memory or planning, and it is used only qualitatively (App. D), since p, q and k are never estimated from EVA's runs.
Questions for authors
  1. Which backbone does Harness VLA use in Table 2? Can you compare all agentic methods on the same backbone and restate the abstract claims accordingly?
  2. Does recovery ever reset object or environment state in simulation? Do the monitor or verifier read simulator-internal state, and which executors are used on each benchmark?
  3. Can you evaluate on the full 50 episodes per task, or over multiple seeds, and report latency and tokens over all episodes?
  4. Can you report real-robot success rates over N trials for VLA-only vs. EVA?
  5. What does EVA do that RoboClaw or Harness VLA cannot, beyond combining their components?
4 comments
AuthorsOpenReview ↗

Response to Reviewer dccg (Part 1).

We sincerely thank you for detailed feedback and we address your concerns as follows.

W1 & Q1: The headline claims compare across backbones.

Thanks for the careful assessment. We first clarify that HarnessVLA in Table 2 uses πXR1\pi_{\mathrm{XR1}}, the same VLA backbone as EVA (πXR1\pi_{\mathrm{XR1}}). We choose πXR1\pi_{\mathrm{XR1}} because it is the strongest VLA baseline on the VLABench, while further evaluating EVA with π0.5\pi_{0.5} allows us to examine whether our harness also benefits a weaker underlying policy. To further ensure the completeness of the experiments, we provide the results for π0.5\pi_{0.5}, as shown in the table below.

Table 1. Results with different VLA backbones on VLABench.

MethodVLA BackboneIn-dist. / SR ↑\uparrowIn-dist. / PS ↑\uparrowIn-dist. / IS ↑\uparrowCross Category / SR ↑\uparrowCross Category / PS ↑\uparrowCross Category / IS ↑\uparrowCommonsense / SR ↑\uparrowCommonsense / PS ↑\uparrowCommonsense / IS ↑\uparrowInstruction / SR ↑\uparrowInstruction / PS ↑\uparrowInstruction / IS ↑\uparrowTexture / SR ↑\uparrowTexture / PS ↑\uparrowTexture / IS ↑\uparrowAverage / SR ↑\uparrowAverage / PS ↑\uparrowAverage / IS ↑\uparrow
VLAπXR1\pi_{\mathrm{XR1}}79.0‾\underline{79.0}87.9‾\underline{87.9}79.8‾\underline{79.8}57.0‾\underline{57.0}69.7‾\underline{69.7}68.461.0‾\underline{61.0}66.5‾\underline{66.5}65.7‾\underline{65.7}67.0‾\underline{67.0}74.2‾\underline{74.2}73.065.076.677.865.8‾\underline{65.8}75.0‾\underline{75.0}72.9
Harness VLAπXR1\pi_{\mathrm{XR1}}41.061.281.034.052.173.432.046.365.038.056.276.0‾\underline{76.0}32.052.575.0‾\underline{75.0}35.453.774.1‾\underline{74.1}
EVAπXR1\pi_{\mathrm{XR1}}84.091.779.665.074.071.0‾\underline{71.0}77.083.278.076.086.581.064.0‾\underline{64.0}75.7‾\underline{75.7}75.0‾\underline{75.0}73.282.276.9
VLAπ0.5\pi_{0.5}43.0‾\underline{43.0}61.1‾\underline{61.1}75.826.0‾\underline{26.0}42.9‾\underline{42.9}70.7‾\underline{70.7}26.0‾\underline{26.0}45.4‾\underline{45.4}66.3‾\underline{66.3}21.0‾\underline{21.0}40.3‾\underline{40.3}67.7‾\underline{67.7}31.0‾\underline{31.0}49.3‾\underline{49.3}70.5‾\underline{70.5}29.4‾\underline{29.4}47.8‾\underline{47.8}70.2
Harness VLAπ0.5\pi_{0.5}28.051.278.0‾\underline{78.0}20.039.573.024.042.366.016.035.366.026.045.876.022.842.871.8‾\underline{71.8}
EVAπ0.5\pi_{0.5}63.076.183.838.056.268.043.061.678.840.062.484.532.053.469.743.261.977.0

As shown above, EVA improves the success rate from 65.8% to 73.2% with πXR1\pi_{\mathrm{XR1}}, and from 29.4% to 43.2% with π0.5\pi_{0.5}, corresponding to relative gains of 11.2% and 47.0%, respectively. Although the absolute performance of π0.5\pi_{0.5} remains lower due to its weaker underlying policy, the relative improvement is substantial. These results demonstrate the effectiveness of our framework across different VLA backbones, while achieving substantially lower token consumption and latency than HarnessVLA.

W2 & Q2: Recovery may use privileged simulator resets.

Thanks for your concern. We would like to clarify that recovery does not reset the simulator or restore object/environment states. The recovery process moves the robot toward a previously recorded robot state via an executed joint-space trajectory, with Ruckig [1] used for trajectory generation. Therefore, this entire procedure does not require an oracle derived from simulator snapshots.

In addition, any object displacement induced by actions or recovery during robot motion persists within the current scene. Therefore, recovery in EVA can facilitate subsequent attempts, yet it cannot undo arbitrary manipulation failures or guarantee restoration of the original scene. This remains a challenging problem for the community, which will be solved in the future.

Regarding the monitor and verifier: neither the monitor nor the semantic verifier accesses hidden object poses, simulator success flags, or oracle task-completion labels. The external success evaluator of this benchmark is also separate from the verifier adopted by the agent, which relies only on observable visual and proprioceptive signals. As for the executors, we use Xiaomi-Robotics-1 or π0.5\pi_{0.5} on VLABench, RLinf-released π0.5\pi_{0.5} SFT checkpoint for full-shot LIBERO-130 on LIBERO-LoHo, and the GroundSG variant of π0.5\pi_{0.5} on RoboMME.

AuthorsOpenReview ↗

Response to Reviewer dccg (Part 2).

W3 & Q3: Evaluation is small and mixed. The evaluation uses only 10 episodes per task and one seed, while copied VLA results use 50. Can evaluation be expanded and costs be reported over all episodes?

Thank you for this valuable suggestion. We clarify that, to ensure a practical comparison, we follow prior work setting [2] and use 10 episodes per task setting for simulation benchmarks, as agentic harnesses typically incur substantially higher inference and execution costs than standalone VLA policies, particularly for approaches that integrate Codex or OpenClaw [2,3]. To assess the reliability of our conclusions under the existing evaluation budget, we compute the 95% Wilson confidence interval on VLABench, which quantifies the uncertainty in estimated task-level success rates, based on 10 episodes per task. The resulting interval on success rate is [69.15[69.15%, 76.89%], showing that the uncertainty in the estimated success rates does not alter our key performance comparisons or the overall conclusions. Nevertheless, we acknowledge that relatively small sample size may introduce variance. To address this concern, we additionally conduct extended evaluation on RoboMME with full 50 episodes per task.

Furthermore, we also agree that reporting average latency and token consumption conditioned on successful completions may involve different subsets of episodes due to differences in success rates. Without explicit execution limits, repeated retries on unsuccessful episodes can lead to unbounded time and token consumption. We therefore impose the same environment-step and token budgets on all methods within each benchmark, and report average latency and token consumption over all episodes, including both successes and failures. The detailed results are presented below.

Table 2. RoboMME results (success rate, %). The best and second-best results in each column are highlighted in bold and underline‾\underline{underline}, respectively.

MethodBin FillPick XtimesSwing XtimesStop CubeVideo UmskButton UmskVideo UmskSButton UmskSPick HighLVideo RepickVideo PlcBtnVideo PlcOrdMove CubeInsert PegPattern LockRoute StickAVG
DeepSeek-V4.1-Flash + GroundSG18.0054.0042.00‾\underline{42.00}6.0076.0022.0034.002.0032.0026.00‾\underline{26.00}28.0034.0048.00‾\underline{48.00}4.00‾\underline{4.00}6.006.0027.38
Qwen3.8-Max + GroundSG42.00‾\underline{42.00}72.0030.008.00‾\underline{8.00}42.0038.0024.002.0054.0014.0020.0036.0052.004.00‾\underline{4.00}8.002.0028.00
Harness VLA + GroundSG32.0082.00‾\underline{82.00}0.0066.0096.0084.0036.00‾\underline{36.00}44.0080.0026.00‾\underline{26.00}54.0082.0042.002.0020.00‾\underline{20.00}14.00‾\underline{14.00}47.50‾\underline{47.50}
EVA + GroundSG76.0094.0080.004.0086.00‾\underline{86.00}66.00‾\underline{66.00}74.0032.00‾\underline{32.00}70.00‾\underline{70.00}78.0044.00‾\underline{44.00}46.00‾\underline{46.00}46.006.0050.0078.0058.13

Table 3. Performance and inference costs of agentic methods on VLABench and LIBERO-LoHo. Latency and token consumption are averaged over all episodes. The best and second-best results within each benchmark are highlighted in bold and underline‾\underline{underline}, respectively.

BenchmarkMethodSR (%) ↑\uparrowLatency (s) ↓\downarrowTokens (K) ↓\downarrow
VLABenchHarness VLA (πXR1\pi_{\mathrm{XR1}})35.4‾\underline{35.4}624.9‾\underline{624.9}1449.6‾\underline{1449.6}
VLABenchEVA (πXR1\pi_{\mathrm{XR1}})73.2319.482.8
LIBERO-LoHoHarness VLA (πRLinf\pi_{\mathrm{RLinf}})34.0‾\underline{34.0}731.98565.3
LIBERO-LoHoVoLo (πRLinf\pi_{\mathrm{RLinf}})27.06563.8418.8‾\underline{418.8}
LIBERO-LoHoEVA (πRLinf\pi_{\mathrm{RLinf}})79.0773.8‾\underline{773.8}220.7

Overall, the expanded evaluation with 50 episodes per task validates the effectiveness of our approach, leading a 10.63 point improvement in success rate, and demonstrates that the observed gains are not merely attributable to evaluation randomness. Moreover, our method maintains its performance advantages while reducing inference latency and token consumption, when failed episodes are taken into account. These results provide further evidence of the effectiveness, efficiency, and robustness of our framework.

AuthorsOpenReview ↗

Response to Reviewer dccg (Part 3).

W4 & Q4: The real-robot result is anecdotal. It is a single qualitative example with no success rate or trial count, yet the abstract presents it as evidence of effectiveness.

Thank you for the valuable suggestion. To provide a quantitative evaluation in real-world settings, we conduct experiments on two real-robot tasks: (1) Three-Cup Monte (hidden-target tracking) task, in which an object is hidden under one of three inverted cups, the cups are shuffled, and the robot must identify the cup containing the object and press the corresponding button. This task evaluates the robot's ability to maintain memory and make correct decisions under visual uncertainty; and (2) Box-packing task, which requires the robot to grasp and lift a box from one table, plan its movement, and place it on another table, assessing its multi-step manipulation and task coordination capabilities.

For both tasks, we compare two settings: (1) Ψ0\Psi_0 (VLA-only), which directly executes the underlying VLA policy Ψ0\Psi_0, and (2) EVA + Ψ0\Psi_0 (Ours), which integrates the same VLA policy into the EVA framework. We use DeepSeek-v4.1-Flash as the underlying Planner and Verifier model in both settings. To ensure a fair comparison, the underlying VLA policy remains unchanged, allowing us to isolate the contribution of the EVA framework beyond the capabilities of the base VLA. For each setting, we conduct 10 independent trials and report the success rate. As shown in the table below, the results demonstrate that integrating EVA with the same underlying VLA policy improves real-world task execution performance across tasks, highlighting the effectiveness of EVA in enhancing embodied interaction.

Table 4. Real-robot task success rate (%) of the VLA-only baseline and EVA with the same VLA policy.

MethodThree-Cup MonteBox PackingAverage
Ψ0\Psi_0 (VLA-only)0/100/100%
EVA + Ψ0\Psi_0 (Ours)7/104/1055%

W5(a) & Q5: What is new beyond RoboClaw, ABot-Claw, Harness VLA, and VoLo?

Thanks for giving the chance for clarification. The novelty of EVA is not an isolated planner, memory module, or verifier. It is an embodied execution protocol that gives each subtask explicit transition semantics. For a subtask zz, EVA specifies not only how it is executed, but also when execution should stop, how completion is verified, how task and object state are updated, and how the system should continue or recover after failure. This makes “the rollout has stopped,” “the subtask has succeeded,” and “the robot is ready for the next subtask” three distinct states, rather than treating them as one event.

This perspective distinguishes EVA from the compared systems. RoboClaw and ABot-Claw primarily organize high-level planning and persistent memory, but do not formulate this explicit execution–termination–verification–recovery transition around every subtask. Harness VLA and Volo use agentic memory to steer a frozen VLA, but does not provide EVA’s separation between high-frequency online termination and post-execution semantic verification, nor its object-centric state update in the execution loop.

Concretely, EVA implements this protocol through (i) task memory for progress and verified outcomes, (ii) object memory for persistent identities and changing spatial states, (iii) event-driven monitoring that terminates an execution when a physical condition is met, (iv) semantic verification after termination, and (v) state-aware replanning or physically valid recovery. These mechanisms are coupled: verification updates memory, memory determines the next subtask and its termination conditions, and recovery changes the robot state without resetting the scene. The contribution is therefore the system-level abstraction and its concrete realization for embodied interaction, not the claim that each component is individually new.

AuthorsOpenReview ↗

Response to Reviewer dccg (Part 4).

W5(b): The theory does not involve memory or planning, and it is used only qualitatively (App. D), since p, q and k are never estimated from EVA's runs.

We agree that the theory does not explicitly model memory or high-level planning. Crucially, it formalizes how EVA's executor, monitor, verifier, and recovery module work together to turn failed attempts into additional opportunities for success, and establishes when this organization improves reliability over single-shot execution. To ground this analysis empirically, we analyzed EVA runs with π0.5\pi_{0.5} on all 250 official configurations of VLABench's select_fruit task. Using simulator ground truth solely for offline measurement, we estimate first-attempt physical success as p=0.5838p=0.5838. We generalize the original paper's detection reliability qq to an asymmetric pair: α=P(accept∣success)=0.9630\alpha=P(\mathrm{accept}\mid\mathrm{success})=0.9630 is the probability of correctly accepting a successful attempt, and β=P(reject∣failure)=0.9091\beta=P(\mathrm{reject}\mid\mathrm{failure})=0.9091 is the probability of correctly rejecting a failed attempt. The original symmetric model is recovered when α=β=q\alpha=\beta=q. We estimate k=0.7500k=0.7500 from the fraction of recovery outcomes with successful robot motion and a task state still eligible for retry, accounting for task-relevant damage from execution and recovery. Substituting these measured parameters directly into Proposition 5.4's finite-retry expression, SR=pα∑j=0R[k(p(1−α)+(1−p)β)]jS_R=p\alpha\sum_{j=0}^{R}[k(p(1-\alpha)+(1-p)\beta)]^j, yields the following comparison with experimental true-and-verifier-accepted pick success rates, without curve-fitting optimization.

Table 5. Predicted and experimental pick-subtask success rates (%), using measured parameters. RR denotes retries after the initial attempt.

Retries RR01234
Formula SRS_R56.2273.0878.1479.6680.11
Experiment56.2267.5769.1970.2770.27

The experimental results follow the predicted trend: retries improve SR by 14.05 percentage points, with diminishing gains as the retry budget increases. The 9.84-point gap at R=4R=4 reflects limitations of the stationary-retry approximation: later attempts involve harder cases, and persistent scene changes can reduce success even after an operationally valid recovery. This quantitative comparison supports the theory's central insight that reliable verification and recovery make EVA's coordinated execution loop effective, while identifying where real-world dynamics depart from the idealized model. Beyond explaining the effectiveness of EVA's organization, the theory provides actionable guidance on which components to improve—executor accuracy, verifier capability, and recovery reliability—and how their joint quality determines the benefit of additional attempts.

Reference:

[1] Berscheid and Kröger, “Jerk-limited Real-time Trajectory Generation with Arbitrary Target States,” RSS, 2021.

[2] Zhang, Yixian, et al. "Harness vla: Steering frozen vlas into reliable manipulation primitives via memory-guided agents." arXiv preprint arXiv:2607.08448 (2026).

[3] Huo, Dongjie, et al. "ABot-Claw: A foundation for persistent, cooperative, and self-evolving robotic agents." arXiv preprint arXiv:2604.10096 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.