- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve the model-external harness of a frozen embodied agent. The harness consists of skills, context selection, verification, and recovery. A frozen vision-language model diagnoses failed trajectories and proposes bounded revisions to individual components. The main mechanism, anchored counterfactual replay, restores simulator state and the recorded history near an observable deviation, compares candidate repairs through paired suffix rollouts, and selects a revision for further validation. A paired acceptance gate requires a positive observed success gain without losing parent-solved validation episodes, together with cost and catastrophic-event constraints. The paper evaluates the framework on VLABench, ESI-Bench, and the Habitat and Navigation subsets of EmbodiedBench. Reported gains over the corresponding frozen agents include 34.8% to 48.9% on VLABench and 39.8% to 62.8% micro accuracy on ESI-Bench. Ablations examine diagnosis, replay, state anchoring, validation, and the editable component families.
Critical strengths
- The paper addresses a meaningful component-selection problem with a concrete testing mechanism. Improving a frozen embodied agent requires deciding which part of its harness to revise, in addition to generating the revision itself. Anchored replay compares candidate edits from the same restored state and history, providing empirical evidence for this decision.
- The evaluation includes useful evidence beyond aggregate performance gains. Experiments cover manipulation, spatial reasoning, and navigation, while the ablations examine diagnosis, replay, state anchoring, and validation. For example, Table 4 reports 54.5% success for the full method versus 50.6% without repair probes, and a lower test regression rate with validation than without it. Figures 5–6 also show a candidate being rejected despite repairing one episode because it loses a parent success. These analyses provide useful supporting evidence for the proposed design and help explain how the update process behaves.
Critical weaknesses
-
The advantage under a matched total optimization budget is not yet sufficiently established. Section 3.3 states that replay costs are charged to the method and competing approaches receive the same total budget. However, Table 4 mainly fixes the optimization schedule, while Appendix E describes different cost treatments for some variants. Reporting total simulator steps and model calls/tokens would help establish whether the gains reflect better allocation of the same resources. A sequential-selection control using the same initial harness, revision generator, validation gate, and total budget would also provide a more direct comparison than published baselines with potentially different development settings. Independent-run counts and uncertainty estimates would strengthen the interpretation of the relatively modest gains from replay and anchoring.
-
The connection between the artifact specification and the implementation examples needs clarification. Appendix A.3 describes skills as structured JSON rules, whereas Figure 6 presents the candidate as natural-language instructions. These may be compatible representations, but the actual serialized artifact and its mapping to planner inputs or executable behavior would make the intervention easier to understand and reproduce. Figures 7–8 explicitly illustrate acceptance of a multi-component bundle; clarifying how this example relates to the single-component procedure used for the aggregate results would further improve the implementation account.
Questions for authors
- Could you report the total optimization cost for each Table 4 variant, including model calls/tokens, simulator steps, and the cost of rejected candidates and validation? Under the same total budget, how does the full method compare with diagnosis-only and sequential component selection using the same initial harness, proposal generator, and validation gate? Please also provide independent-run counts and uncertainty estimates.
- Could you provide the parent and candidate artifacts corresponding to Figure 6 and explain how they are converted into planner inputs or executable rules? This would help connect the structured JSON interface in Appendix A.3 with the natural-language representation shown in the figure.
- Please clarify whether Tables 1–4 use the single-component update procedure in Algorithm 1, and how the multi-component bundle example in Figures 7–8 relates to those experiments. An accepted single-component example with its replay and validation outcomes would help illustrate the complete procedure.
3 comments
W2. "The connection between the artifact specification and the implementation examples needs clarification. Appendix A.3 describes skills as structured JSON rules, whereas Figure 6 presents the candidate as natural-language instructions. [...] Figures 7-8 explicitly illustrate acceptance of a multi-component bundle."
Thank you for pointing out this inconsistency. We had not clearly distinguished the artifact's storage format from its content. Revised Appendix A.3 and the Figure 6 caption now state the following:
- A skill is stored as a JSON record of named rules, each with a trigger and an optional precondition.
- Its content is natural-language guidance.
- At runtime, this guidance is inserted into the frozen planner's prompt, and the planner issues requests through the existing action interface.
- The text is not parsed into deterministic executable rules.
- Context, verification and recovery, by contrast, set structured fields that the runtime enforces directly.
The complete serialized parent and candidate artifacts, with their diff, are in the supplementary material.
Figure 6 parent versus candidate.
- Parent: instructs the agent to identify the requested object from observations, pick and place it, and re-observe uncertain outcomes.
- Candidate: adds guidance for selecting objects by semantic attributes, recommends placing immediately after picking, and discourages subsequent lifting.
Only the skill changes. This repairs the cola task, but a previously solved playing-card task that requires a lift then fails. Anchored replay gives a net gain of +0.2 over five complete pairs. The canary then records one repair and one regression (W = L = 1), so the gate rejects the update. Replay selects promising repairs; validation checks whether they harm other tasks.
Figures 7–8 are a multi-component case, and the revision now says so explicitly.
- They change skill, verification and recovery together.
- On six reachable development episodes, success rises from 2 to 3 with no parent-solved episode lost, so the update passes the gate.
- App. F (intro, Fig. 7 caption and F.4) now labels this as a multi-component bundle that shows how the gate treats such a candidate, like the edit-all control. It is not an instance of Algorithm 1.
Q1. Could you report the total optimization cost for each Table 4 variant, including model calls/tokens, simulator steps, and the cost of rejected candidates and validation? Under the same total budget, how does the full method compare with diagnosis-only and sequential component selection using the same initial harness, proposal generator, and validation gate? Please also provide independent-run counts and uncertainty estimates.
Cost per variant. Table R1 (revised Table 8) gives calls, tokens, simulator steps and hours for all nine variants. Rejected candidates and validation are included in their stages (Table 9).
Matched total budget. Table R2 (under W1) answers this directly. Under common token and simulator-step caps, with the same initial harness, proposal generator and gate, the full method reaches 54.5 ± 0.55%. Diagnosis-only reaches 52.1 ± 0.85% (−2.4 pp, paired 95% CI [−3.39, −1.41]) and sequential component selection 50.7 ± 0.95% (−3.8 pp, [−4.50, −3.10]).
Run counts and uncertainty. Table R2 uses five independent runs per method and reports the SD across runs and paired 95% CIs. The restoration-noise study averages three runs (App. G.2) and the retention study five (App. G.3).
Q2. Could you provide the parent and candidate artifacts corresponding to Figure 6 and explain how they are converted into planner inputs or executable rules? This would help connect the structured JSON interface in Appendix A.3 with the natural-language representation shown in the figure.
Artifacts. Both artifacts are JSON records whose rule content is natural-language guidance. The guidance is supplied to the planner as prompt text, and the planner acts through the existing execution interface. It is not converted into executable rules.
What the revision provides. Revised App. A.3 and the Fig. 6 caption explain this. The supplementary material contains both complete serialized artifacts and their diff.
Q3. Please clarify whether Tables 1–4 use the single-component update procedure in Algorithm 1, and how the multi-component bundle example in Figures 7–8 relates to those experiments. An accepted single-component example with its replay and validation outcomes would help illustrate the complete procedure.
Tables 1–4. We clarify here that the EmbodiedHarness rows of Tables 1–4 use Algorithm 1, the default single-component procedure. In Table 4, edit-all and no-gate are the explicit multi-component and ungated controls.
Figures 7–8. Figures 7–8 illustrate the gate on a multi-component bundle, labelled as such in App. F.
Accepted single-component example. Revised App. G.5 (Table 13, top) traces a VLABench run whose three accepted revisions are all single-component promotions:
- skill: 2 → 3 rules;
- context: 3 → 5 recent views;
- verifier: disabled → at most one review call.
We hope these additions address your concerns and welcome further discussion.
Thank you for the careful review. We address all your questions and concerns below.
W1. "The advantage under a matched total optimization budget is not yet sufficiently established. [...] Reporting total simulator steps and model calls/tokens would help [...]. A sequential-selection control [...] would also provide a more direct comparison. Independent-run counts and uncertainty estimates would strengthen the interpretation."
Equal rounds do not imply equal cost. The ablations of Table 4 hold the optimization schedule, not the total cost, fixed; Appendix E no longer calls the no-gate row cost-matched. New App. G.1 (Table 8) reports the acquisition cost per run of every Table 4 variant, covering trajectory collection, diagnosis and proposals, replay, rejected candidates, canary and validation, and retries (sealed-test evaluation and deployment inference excluded).
Table R1 (= revised Table 8). Table 4 outcomes and cost per run (VLABench).
What the cost accounting shows.
Matched total budget (App. G.1). All methods share the initial harness, backbone, proposal generator, data, paired seeds, edit limits and gate, under common caps of 29.2M tokens and 510k simulator steps; budget a control saves funds further attempts. Diagnosis-only uses no probes, sequential selection cycles through the four components in a fixed order, and no anchoring replays from reset. Each method is run five times; run i of every method uses the same seeds and evaluation manifest, and CIs are paired t-intervals (df = 4).
Table R2 (= revised Table 10). Matched total-budget comparison on VLABench (five runs per method).
What Table R2 shows. All controls use 28.8–29.1M tokens and 485–505k steps, and none of the paired CIs includes zero.
Other run counts. The restoration-noise study averages three runs (App. G.2) and the twelve-round retention study five (App. G.3). There, the full method's VLABench mean is 48.7% at round 4 and 48.9% at round 12, consistent with the 48.9% sealed-test value of Table 1.