- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes EMbodied Agent Harness Optimization (EMHO), a framework that automatically optimizes the embodied agent harness using execution traces and historical harness modifications while keeping the underlying model frozen. EMHO iteratively analyzes task execution experiences, modifies harness components such as planning, memory, tool usage, and failure recovery, and retains modifications that improve task success rates. The paper further introduces EMHO-Merge, which uses episode-level gains and losses to refine a shared harness across multiple subtasks and mitigate performance regressions. Experiments on EmbodiedBench across navigation and manipulation tasks demonstrate improvements over the initial harness for both Qwen 9B and 27B models, with additional comparisons against one-shot and skill-only optimization baselines.
Critical strengths
The paper addresses an important and meaningful research problem: automatically optimizing the entire embodied agent harness rather than relying on model training or optimizing individual components. This research direction is well-motivated and has significant potential for improving embodied agents.
Critical weaknesses
-
EMHO repeatedly uses the same Search Set throughout the optimization process, making it prone to overfitting and lacking effective mechanisms to ensure generalization.
-
EMHO generates only one candidate from the current best Harness in each iteration and accepts the modification only if the candidate achieves a strictly higher success rate on the Search Set. This search strategy limits exploration and may prematurely discard promising optimization directions.
-
EMHO relies on continuously accumulated experience traces and harness modification histories for optimization. However, large amounts of redundant, outdated, or conflicting information may interfere with the optimizer's decision-making and increase computational overhead.
-
EMHO heavily relies on an LLM to perform a series of tasks, including trajectory analysis, failure diagnosis, modification planning, and Harness code editing, making the process susceptible to hallucinations and error propagation. Moreover, Harness editing operations are primarily restricted through prompt-based soft constraints on the editable scope, which are unreliable.
-
Although the paper emphasizes that EMHO can optimize the entire Harness, it does not systematically investigate or evaluate the evolution of workflow structures, such as component execution order and stage nesting. Therefore, its claim of full-Harness optimization is insufficiently substantiated.
-
Table 1 reports the best Search Set and Held-out Set results separately across optimization iterations, while Appendix B further acknowledges that these results may come from different iterations. Therefore, the reported best Held-out performance does not necessarily correspond to the final harness selected based on Search Set performance and cannot accurately reflect the generalization performance of EMHO.
Questions for authors
-
Table 3 does not report the Held-out results of EMHO (Combined), making it difficult to assess whether EMHO-Merge improves generalization over standard shared-harness optimization. Moreover, EMHO-Merge performs worse than EMHO (Separate) on both the manipulation Search and Held-out sets. Could the authors provide the missing results and clarify the actual benefits of EMHO-Merge?
-
Table 2 compares full-harness optimization with skill-only optimization exclusively on the Search Set, without reporting Held-out performance. Moreover, full-harness optimization does not consistently outperform skill-only optimization across subtasks. Could the authors provide Held-out comparisons to substantiate the claimed advantages of full-harness optimization?
-
EMHO performs 10 iterations of experience collection, harness revision, and evaluation, whereas the one-shot baselines are allowed only a single revision. This gives EMHO substantially more opportunities to exploit execution feedback and improve performance. Does EMHO still outperform these baselines under matched optimization and task-execution budgets?