- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper develops a harness built around different types of memory to improve manipulation performance. It also augments the VLA with additional capabilities supported by foundation models, ultimately achieving substantial improvements on several long-horizon tasks.
Critical strengths
- The paper achieves solid performance improvements and includes supporting real-world robot experiments.
- The overall narrative is clear, making the key aspects of the design easy to follow.
Critical weaknesses
- The paper lacks visual examples, including visualizations of execution processes in the experiments and diagrams that facilitate understanding and comparison.
- The paper does not include comparisons on a broader range of commonly used benchmarks.
- The harness designs are fairly common. More comprehensive comparisons with similar designs in other agentic approaches are needed to understand where the final performance gains come from.
Questions for authors
- The designs of the individual harness components are fairly common. How does the proposed approach differ from similar work in its design, and where do the final performance gains come from?
- Should results on more commonly used benchmarks be included? Even for a method designed for long-horizon tasks, performance on general-purpose tasks remains important.
- What additional overhead does the harness introduce compared with end-to-end models? This warrants a dedicated analysis. Whether the harness yields different gains when paired with different underlying models also requires thorough examination.
2 comments
Response to Reviewer pBxX (Part 2).
Q3(a): What overhead does the harness introduce relative to an end-to-end model?
Thank you for this question. As analyzed in Appendix D (Deployment Timeline Analysis, Figure 5), EVA introduces additional latency primarily at subtask transitions, mainly from high-level planning and outcome verification, while checkpoint recovery is comparatively short. Importantly, EVA mitigates this overhead through a lightweight monitor that triggers semantic verification only when execution termination is detected, rather than at every control step. Furthermore, perception is processed asynchronously to overlap with other stages where possible. Thus, EVA concentrates additional reasoning overhead at critical decision boundaries while minimizing unnecessary synchronization, enabling more reliable long-horizon execution without repeatedly invoking expensive high-level models.
Q3(b): Does the benefit depend on the underlying models?
We investigate the dependence of EVA's benefits on the underlying models from two perspectives: (1) low-level VLA policy and (2) high-level planning and verification models.
For low-level VLA policy, as shown in Table 3, on VLABench, the success rate improves from 29.4% to 43.2% with and from 65.8% to 73.2% with . These results demonstrate that EVA is applicable to different low-level VLA policies with various improvement.
As for high-level planning and verifier, We further conduct additional experiments on LIBERO-LoHo by varying the underlying models used for planning and verification. Specifically, we compare three model configurations: Qwen3.8-27B (a smaller open-source model), DeepSeek-V4.1-Flash (a larger open-source model), and GPT-6-Astra (a frontier closed-source model). The results are reported in the table below.
Table 7. Performance of EVA with different LLM backbones on LIBERO-LoHo. We report Q-Score (QS, %) and success rate (SR, %). The best and second-best results in each column are highlighted in bold and , respectively.
| LLM Backbone | Task 1 / QS | Task 1 / SR | Task 2 / QS | Task 2 / SR | Task 3 / QS | Task 3 / SR | Task 4 / QS | Task 4 / SR | Task 5 / QS | Task 5 / SR | Average / QS | Average / SR |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B | 88.3 | 78.3 | 55.0 | 81.7 | 96.7 | 90.0 | 80.0 | 35.0 | 85.0 | |||
| DeepSeek-V4.1-Flash | 90.0 | 95.0 | 85.0 | 91.7 | 75.0 | 90.0 | 91.7 | 77.0 | ||||
| GPT-6-Astra | 95.0 | 90.0 | 75.0 | 90.0 | 85.0 | 65.0 | 77.0 |
The results show that the Qwen-based configuration achieves a 61% success rate, while the DeepSeek- and Astra-based configurations achieve 77%. This demonstrates that EVA maintains good performance even when using a relatively small model such as Qwen3.8-27B. Moreover, we observe no significant performance difference between the two stronger models and no single model consistently dominates across all tasks.
To better understand the performance, we further conduct trajectory analysis. Our analysis suggests that the lower success rate of Qwen3.8-27B is primarily associated with less reliable completion-state verification and recovery planning. In particular, these models are more likely to incorrectly assess whether a task has been completed or fail to generate effective recovery plans when execution deviates from expectations. This suggests that improving VLMs' capabilities in completion-state verification and recovery planning could further enhance the overall system performance.
Overall, these results suggest that EVA's benefits are not limited to a particular underlying model. While the absolute performance may vary with model capability, EVA remains effective across different low-level policies and high-level planning and verification models.
Reference:
[1] Zhou et al., “LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization,” arXiv, 2025.
Response to Reviewer pBxX (Part 1).
We would like to express our sincere gratitude for your valuable feedback and comments and we address your concerns as follows.
W1: The paper lacks execution visualizations and diagrams that facilitate understanding and comparison.
Thank you for this insightful suggestion. We will provide a dedicated demo page https://eva-anonymous.github.io/ featuring visual demonstrations of the effectiveness of each component, simulation performance comparisons between EVA and other baselines, and real-world robot experiments. These demonstrations will offer a more intuitive understanding of EVA's component-level contributions, comparative performance, and real-world applicability.
W2 & Q2: Should results on more commonly used benchmarks be included? Even for a method designed for long-horizon tasks, performance on general-purpose tasks remains important
We agree that evaluating general-purpose performance is important. In fact, our existing evaluations on VLABench, RoboMME, and LIBERO-LoHo already cover a range of general-purpose manipulation tasks like picking and placing objects, pressing buttons, opening and closing drawers, inserting flowers, and adding condiment. For instance, VLABench includes diverse general-purpose manipulation tasks and evaluates robustness beyond individual long-horizon sequences. The results on these benchmarks already demonstrate EVA's strong performance, indicating its capability to handle general-purpose manipulation tasks effectively.
To further address the reviewer's concern regarding performance on general-purpose benchmarks, we additionally evaluate EVA on LIBERO-PRO[1] benchmark, covering four suites with 10 tasks each under five perturbation types. We instantiate EVA with π0 VLA policy and compare EVA with raw π0.5 policy to further evaluate its improvement on general-purpose tasks. We use initial state 0-1 per task–perturbation pair (seed 0), yielding 400 rollouts per method. The results are presented below.
Table 6. Results on LIBERO-Pro (success rate, %).
The results show that EVA with π0 outperforms the raw π0 baseline across three perturbation types object, language and task on LIBERO-PRO, achieving an relative success rate improvement of 25.8%, confirming that our framework not only excels in long-horizon tasks but also enhances general-purpose manipulation capabilities in different benchmarks.
W3 & Q1: How does EVA differ from similar harness designs, and where do the gains come from?
Thanks for giving the chance for clarification. The novelty of EVA is not an isolated planner, memory module, or verifier. It is an embodied execution protocol that gives each subtask explicit transition semantics. For a subtask z, EVA specifies not only how it is executed, but also when execution should stop, how completion is verified, how task and object state are updated, and how the system should continue or recover after failure. This makes “the rollout has stopped,” “the subtask has succeeded,” and “the robot is ready for the next subtask” three distinct states, rather than treating them as one event.
This perspective distinguishes EVA from the compared systems. RoboClaw and ABot-Claw primarily organize high-level planning and persistent memory, but do not formulate this explicit execution–termination–verification–recovery transition around every subtask. Harness VLA and Volo use agentic memory to steer a frozen VLA, but does not provide EVA’s separation between high-frequency online termination and post-execution semantic verification, nor its object-centric state update in the execution loop.
Concretely, EVA implements this protocol through (i) task memory for progress and verified outcomes, (ii) object memory for persistent identities and changing spatial states, (iii) event-driven monitoring that terminates an execution when a physical condition is met, (iv) semantic verification after termination, and (v) state-aware replanning or physically valid recovery. These mechanisms are coupled: verification updates memory, memory determines the next subtask and its termination conditions, and recovery changes the robot state without resetting the scene. The contribution is therefore the system-level abstraction and its concrete realization for embodied interaction, not the claim that each component is individually new. Ablation studies in Appendix D further demonstrates the effectiveness of these design choices from an empirical perspective.