- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EnactOpt, a framework that adapts embodied agents by evolving their execution harness, consisting of reusable textual skills S and context-construction code C, while keeping the foundation models and runtime fixed. For each training rollout, a diagnostic LLM examines adjacent decisions, relating the preceding action feedback to the actual model input x_t and the current action, to judge whether the feedback was presented in the input and whether it was used appropriately; missing evidence suggests revising C, and inadequate responses suggest revising S. These diagnoses are condensed into structured episode summaries, which a single optimizer LLM uses to propose joint revisions of S and C under six prompt-level edit operators (add, revise, delete, move, split, merge), and compatible candidates are retained in a top-k pool according to validation success.
The method is evaluated on VLABench (with a frozen π0 low-level policy), ALFWorld, and EAI VirtualHome action sequencing, using GPT-5.4 mini for all LLM components, 30 training and 30 validation instances per benchmark, and three evolution iterations. EnactOpt improves test success over the initial harness by 10.43, 26.64, and 23.40 percentage points, respectively, and outperforms GEPA and skill-only reimplementations of TextGrad and Trace2Skill on all three benchmarks. Gains also appear on held-out VLABench tracks (T3/T4/T6) and ALFWorld unseen scenes, and VLABench ablations indicate that removing adjacent-decision diagnosis or structured edit operators hurts generalization, while joint revision outperforms separate skill and code updates with half the optimization passes.
Critical strengths
- The paper separates failures in which evidence was missing from the actual model input x_t from failures in which available evidence was misused, which gives a principled rationale for jointly revising skills and context code. This decomposition is the conceptual core that distinguishes the work from outcome-driven prompt optimization and is broadly applicable to agent harness design.
- Improvements hold across a low-level policy benchmark and two high-level action benchmarks, and the VLABench ablation shows that variants with similar in-distribution success can differ substantially under shift (e.g., outcome-only feedback gains 3.75 pp in-distribution but loses 1.91 pp on generalization). This finding, together with the full release of evolved artifacts in Appendix B, is useful for how harness optimization should be evaluated.
Critical weaknesses
- The closest prior work, SHAPER (Wang et al., 2026b), which also evolves skills and context-construction code with diagnosis and preservation-aware edits, is not compared, and TextGrad and Trace2Skill are skill-only reimplementations with C fixed. Since several evolved changes require editing C (e.g., expanding the ALFWorld history window from 2 to 50 turns), the main-table gains partly reflect a larger editable scope rather than the diagnosis or joint-revision design.
- VLABench gains may rely on privileged information and favorable task selection. The evolved context code exposes exact entity identifiers and the track label from observation.extra to the planner, and the evolved skill mainly instructs the planner to emit short canonical commands for the frozen π0, which is consistent with much larger gains under instruction shift (T3/T4, about +16 pp) than under visual shift (T6, +5 pp). Tasks were also selected based on the checkpoint’s reported T1 performance and T1-T4 gaps rather than randomly, so it is unclear how much of the reported generalization reflects harness evolution versus a setting that favors instruction normalization.
- All results come from single evolution runs without variance or confidence intervals, and Figure 3 explicitly uses runs different from Table 1 for some entries, implying run-to-run variation that is not quantified. With only 30 validation instances (3.3 pp per instance) driving selection and 240 T1 test episodes, small differences such as the in-distribution results of EnactOpt and its ablations (63.75% vs. 63.33% vs. 62.08%) cannot be reliably distinguished from noise.
- The paper does not evaluate whether adjacent-decision diagnoses correctly attribute failures to presentation (C) or usage (S), and the w/o-diagnosis ablation only replaces them with terminal outcomes, so it cannot separate the specific adjacent-decision design from simply providing step-level information. In addition, the structured edit operators are unenforced prompt conventions, yet removing them drops performance below H0, which makes it hard to attribute the gains to a well-defined mechanism rather than to prompt wording.
Questions for authors
- What are the total number of environment rollouts and LLM calls (or tokens) used during evolution, including per-candidate validation and per-decision diagnosis, and how many candidates are generated per iteration?
- In EAI, do Naive and all baselines use the same execution feedback loop and attempt budget as EnactOpt?
- Can you report results over multiple evolution seeds, and explain why some Figure 3 entries use runs different from Table 1?