acceptodds
Under review as a conference paper at ICLR 2027

AgentSpec: Evolving Embodied-Agent Harnesses through Typed Composition

Abstract

Embodied LLM agents run inside harnesses of perception, memory, reasoning, reflection, and action, and coding agents can now write and revise these harnesses. We present AgentSpec, a typed specification and runtime in which each component has a declared contract and a complete harness is an executable, diffable program that a coding agent can modify, compose, run, and trace. With it we compare how one coding agent constructs harnesses for one fixed executor: evolving reasoning, memory, and reflection components inside declared contracts over five rounds of execution feedback, writing a free-form program once, and selecting a library configuration, on DeliveryBench, ALFRED, MiniGrid, RoboTHOR, and DeliveryGym. On DeliveryBench the evolved harness, which compresses the action history into an outcome-aligned ledger and adjudicates between sampled decisions, exceeds both baselines, and its advantage persists on average when the frozen harness is re-executed. On MiniGrid the one-shot programs, which decode the rendered observation into a map and plan on it, exceed the evolved harness by a wide margin on the development suite and on held-out tasks. Reading the generated implementations against their execution records shows that, within the editing scope of this study, the two workflows divided work differently between model and program: evolution refined what the executor saw at each step and kept it in the loop, whereas free-form generation moved state maintenance into code and consulted the executor far less often, at the same recorded success rate on ALFRED and a fraction of the cost. AgentSpec makes such divisions of computation inspectable, attributable to specific component changes, and accountable in design, evaluation, and deployment cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, harness evolution, embodied agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nCXQOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

AgentSpec presents a typed specification for embodied LLM-agent harnesses, with explicit interfaces and state lifecycles for perception, memory, reasoning, reflection, and action. In AgentSpec, a separate coding agent (Claude Opus 5 in CC) corrects and proposes changes to reasoning, memory, and reflection components using execution rollouts feedback. An executor model (Qwen 3.5 9B) remains fixed. The paper compares this iterative workflow with fixed human-written library configurations and standalone harnesses generated without execution feedback across DeliveryBench, ALFRED, MiniGrid, RoboTHOR, and DeliveryGym. Overall results vary by environment: the evolved harness outperforms the tested baselines on DeliveryBench. The Freeform baseline programs perform better on MiniGrid, including held-out tasks. The experiments clearly divide what each agent handle directly in code. The paper also carefully splits the cost for building the harness, testing it and running each agent (Proposer vs. Executor).

Critical strengths
  • Explicit component contracts define each module’s inputs, outputs, and state lifecycle (Table 4).
  • Concrete implementation analysis identifies an ALFRED memory bookkeeping error and documents its repair (Appendix B.1).
  • The paper details cost reporting separates the resources used to develop, evaluate, and run each agent (Table 9).
Critical weaknesses
  • Incomplete baseline comparisons. Table 1 lacks full Library-Fixed results for ALFRED and DeliveryGym, and Freeform-f02/f03 results for DeliveryBench. The omission of these full comparisons is not explicitly justified.
  • Insufficient differentiation from prior work. AgentSquare already provides standardized modules and code evolution, while AFlow optimizes executable workflows using evaluation feedback. The additional practical benefits of AgentSpec’s runtime and lifecycle contracts need clearer justification.
  • Difficult presentation. Many method names and run labels are introduced, while important experimental details are scattered across the main text and extensive appendices. This makes the setup difficult to reconstruct. The appendix section is too long to know what to read in priority.
Questions for authors
  • Missing baselines: Could you provide the full Library-Fixed results for ALFRED and DeliveryGym, and the Freeform-f02/f03 results for DeliveryBench, under the same evaluation protocol? Why are these comparisons missing from Table 1?
  • Contribution beyond prior work: Which runtime and state-management features distinguish AgentSpec from AgentSquare and AFlow? Could you demonstrate their practical benefits through a direct comparison or targeted ablations?
8 comments
AuthorsOpenReview ↗

Response to Reviewer nCXQ (1/2)

We thank the reviewer for recognizing the explicit component contracts, the memory diagnosis, and the cost accounting. We added the complete ALFRED library evaluation and the two missing DeliveryBench freeform designs, and clarify the provenance of the empty cells below.


W1/Q1: Missing baselines: Could you provide the full Library-Fixed results for ALFRED and DeliveryGym, and the Freeform-f02/f03 results for DeliveryBench, under the same evaluation protocol? Why are these comparisons missing from Table 1?

A: The gate explains the original missing library cells. In a gated campaign, all three candidates receive gate evaluations, and only the gate winner proceeds to the complete suite. ReAct+Base scored 0.00 on ALFRED's four-task gate and was not selected for full-suite evaluation. DeliveryGym likewise had a gate-only library record. An empty cell denotes a full-suite evaluation that was not performed, rather than a zero score or an omitted completed result.

ALFRED Library-Fixed is now evaluated on the full suite. Three ten-task executions yield 0.20, 0.10, 0.30, or 0.20 ± 0.10. The selected evolved and earlier freeform harnesses average 0.43 ± 0.12 and 0.43 ± 0.06, respectively. This completes the particular fixed-reference comparison and directly addresses the ALFRED request.

The broader library sample is reported separately: among six complete ALFRED configuration evaluations, CoT+Base+Reflexion reaches 0.70 in one execution. This distinguishes performance against the fixed ReAct+Base reference from performance against a searched library maximum. Our response to VSZz gives the sample sizes and distributions for all three environments.

DeliveryBench now includes the two additional current-protocol freeform designs:

Freeform designMean hourly profit
Earlier-protocol program12.44 recorded; 9.0 ± 4.3 across five new executions
Additional current-protocol draw 16.59, one complete-suite execution
Additional current-protocol draw 2−4.00, one complete-suite execution

Both new programs were generated in fresh no-feedback sessions and evaluated on both episodes. The −4.00 program completed evaluation without completing a delivery. The sessions use the study's designer model and effort through an unattended API route with a spending cap; this deployment difference is stated alongside the results. The evolved harness's repeated mean is 15.6 ± 5.8; the full three-harness repetition table, including CoT+MemoryBank, appears in the response to VSZz Q2.

DeliveryGym coverage: ReAct+Base has a gate-only score of 8.33 on four shifts. The full 32-shift library score is unmeasured and labelled accordingly; the gate score is not used in the full-suite comparison.


AuthorsOpenReview ↗

Response to Reviewer nCXQ (2/2)

W2/Q2: Contribution beyond prior work: Which runtime and state-management features distinguish AgentSpec from AgentSquare and AFlow? Could you demonstrate their practical benefits through a direct comparison or targeted ablations?

A: The contribution is the embodied execution contract and the component-level analysis it enables. AgentSquare studies module evolution and recombination; AFlow studies executable workflow search. AgentSpec specifies how generated components participate in a persistent embodied loop: what they receive, what state survives each action, when state resets, and how executed-action outcomes return to memory and reflection.

The practical capabilities are:

Runtime capabilityOperational purposeEvidence in this study
Declared interfaces and state lifecyclesReplace a component while preserving the surrounding loopTable 4; ALFRED memory-only replacement
Explicit edit boundaries and frozen implementationsIdentify which code change a candidate evaluation measuresTable 5; candidate specifications and implementation records
Component traces and cost accountingConnect behavior to state updates and model-call patternsALFRED outcome diagnosis; repeated performance–cost comparison

Targeted component evidence. Appendix B.1 identifies a recency memory that renders an action alongside the previous action's outcome. OutcomeTraceMemV1 uses the transition already supplied to memory to align each recorded action with its own consequence. The reasoner, reflection component, perception, and loop remain unchanged. The campaign scores are 0.30 → 0.50; three re-executions per harness yield 0.37 ± 0.12 → 0.43 ± 0.12.

Answer to the practical-benefit question. AgentSpec makes the state defect visible in a component trace, confines the repair to one replacement unit, and evaluates it with the surrounding harness unchanged. The repeated mean difference is approximately 0.07; the campaign's 0.20 difference is a single-evaluation observation. This demonstrates a concrete runtime capability: inspecting temporal state semantics and testing the corresponding localized repair.

The additional controls assess construction behavior. AgentSpec-OneShot produces a no-feedback DeliveryBench candidate scoring 15.81 and an ALFRED design scoring 0.20. The library samples quantify configuration selection without generated code. These are complementary controls for the studied workflows. Uniform library sampling is not a reproduction of AgentSquare's full algorithm, and the reported study does not include a direct AgentSquare/AFlow performance run.


W3: Difficult presentation. Many method names and run labels are introduced, while important experimental details are scattered across the main text and extensive appendices. This makes the setup difficult to reconstruct. The appendix section is too long to know what to read in priority.

A: The protocol can be read directly through the following distinctions:

WorkflowLibrary accessFeedback during designEditable implementation
Library-FixedHuman-written componentsNoneConfiguration only
Freeform-OneShotNo component-library accessNoneStandalone harness, including observation processing
AgentSpec-OneShotYesNoneReasoning, memory, reflection under contracts
AgentSpec-EvolveYesPrior-round evaluationsThe same component families, three candidates per round, five rounds

The f01/f02/f03 names identify generated programs and protocol versions; the substantive distinctions are earlier-protocol versus current-protocol draws. Section 3 and Tables 4–5 specify contracts and editing scope; Sections 4–5 and Appendix A specify evaluation; Appendix B provides implementation cases. Appendix C is a separate component study with its own model and task settings. The definitions, compact protocol table, and new results in these responses make those distinctions explicit without requiring the reader to reconstruct them from run identifiers.

Thank you for the additional experiments and for being transparent about the corrections. These address most of my baseline concerns.

Reviewer VSZz raised concerns about the reliability of the DeliveryBench improvement. Tte library baseline CoT+MemoryBank now has a higher mean score than the evolved harness (22.1 vs 15.6), it changes one of the paper's central empirical conclusions . Together with the extensive corrections in the rebuttal, this makes it necessary to reassess which conclusions still hold, yet no revised manuscript has been shared and, without code, the numbers cannot be checked. I also agree with Reviewer Wvjz, the paper, and now the rebuttal, are very hard to follow.

I understand that the intended contribution centres on the embodied harness specification and its component analysis. Before finalising my assessment, two clarifications would help:

  1. In the ALFRED example, the buggy memory satisfied its contract, so the contracts did not catch the bug. What did they add in this example compared with ordinary modular code that logs every step? And what exactly does the runtime check or enforce?

  2. Could you provide the per-task outcomes across the three repeated runs of each harness in the memory-only comparison, and explain whether the improvements consistently correspond to the action–outcome alignment defect?

These clarifications would help me judge whether the new data establishes a distinctive and supported runtime contribution.

Reviewer_VSZzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AgentSpec, a typed runtime for evolving embodied LLM-agent harnesses through explicit component contracts and execution feedback. It compares fixed library configurations, one-shot free-form generation, and five-round component evolution using Qwen3.5-9B across five environments. Evolution performs best on DeliveryBench, while one-shot programs outperform it on MiniGrid, including held-out tasks. The paper highlights how editing constraints shape the division of work between models and code.

Critical strengths
  1. Typed contracts improve traceability. AgentSpec makes component states and changes inspectable. In ALFRED, the authors found that the memory module recorded action outcomes one step late. After fixing this issue, unsuccessful terminations dropped from four to one. This is a useful example of how AgentSpec helps identify problems in individual components.
  2. Analysis of performance and cost across workflows. The paper reports candidate histories, frozen-harness repeats, held-out MiniGrid evaluations, and deployment costs, including unfavorable results. Its ALFRED comparison shows that two harnesses can achieve the same 0.50 success rate while using 90 versus 14 executor calls per episode, an informative systems finding.
Critical weaknesses
  1. The comparison does not isolate AgentSpec's contribution. The workflows differ simultaneously in feedback, search budget, library access, and editing scope. Table 6 lists relevant controls, including AgentSpec-OneShot and Freeform-Evolve, but they are not run. Thus, the results establish workflow-level differences rather than the independent benefit of typed evolution. On MiniGrid, the one-shot programs can build maps from images, while evolution cannot modify perception, further confounding the comparison.
  2. Small suites and selection effects weaken the performance claims. DeliveryBench uses two episodes on one map; its best-of-five score of 32.07 falls to 17.8 on three frozen-harness repeats, compared with 13.8 for one-shot. On ALFRED, both methods achieved a success rate of 0.50 on just ten tasks, but no repeated results were reported. It is therefore hard to tell whether their performance is truly comparable. More runs or a comparison under the same evaluation budget would be better.
Questions for authors
  1. What is the maximum over fifteen uniform library draws on the development suite, reported for DeliveryBench, ALFRED and MiniGrid? I appreciate that matched-budget runs are expensive, but without AgentSpec-oneshot or freeform-evolve the paper cannot separate "typed contracts help" from "more attempts help", and that separation is the contribution the paper claims.
  2. The DeliveryBench result seems somewhat unstable. The best score is 32.07, but the same harness only achieves 17.8 on average in repeated runs. Since the evaluation uses just two episodes from one map, could the authors clarify how reliable this improvement is? And since ALFRED carries the cost argument, why is it missing from the repeat table, when a three-repeat of the two 0.50 harnesses is cheap and would settle whether the tie, and the cost claim built on it, is real?
  3. I noticed some inconsistencies in the reported executor calls. For example, DeliveryBench reports 174 calls in Section 6 but 175 in Table 2, while the one-shot agent is reported as 31 and 42 calls, respectively. Similar differences appear in the MiniGrid results. Could the authors clarify why these numbers differ?
6 comments
AuthorsOpenReview ↗

Response to Reviewer VSZz (1/4)

We thank the reviewer for highlighting component traceability and the performance–cost analysis. We added the requested AgentSpec-OneShot control, uniform library sampling, ALFRED repetitions, and an observation-interface intervention. Below we address how these experiments distinguish the construction workflows.


W1/Q1: The comparison does not isolate AgentSpec's contribution. The workflows differ simultaneously in feedback, search budget, library access, and editing scope. Table 6 lists relevant controls, including AgentSpec-OneShot and Freeform-Evolve, but they are not run. Thus, the results establish workflow-level differences rather than the independent benefit of typed evolution.

A: We added a no-feedback component-design workflow under the same contracts. In AgentSpec-OneShot, a fresh designer reads the library, may add reasoning, memory, and reflection implementations within the same write scope, and submits three candidates in one pass. Its first choice is declared before evaluation. The designer receives no execution feedback, cannot run the benchmark, and its implementations are frozen before formal evaluation.

DeliveryBench constructionCompleted full-suite resultsEvaluation unit
AgentSpec-OneShot, session 1−1.39 / 11.27 / 7.14Three candidates; advance first choice −1.39; evaluated maximum 11.27
AgentSpec-OneShot, session 215.81Advance first choice; one completed candidate evaluation reported
Freeform-OneShot, earlier protocol9.0 ± 4.3One frozen program, five executions
Freeform-OneShot, two additional current-protocol sessions6.59 / −4.00One program and one execution per session
AgentSpec-Evolve, selected round-3 harness15.6 ± 5.8One frozen campaign winner, five executions

What the control establishes. Contract-bound component construction can produce a competitive DeliveryBench design without execution-guided iteration: session 2's declared first choice scores 15.81 in one complete-suite evaluation. The experiment therefore supplies the intermediate workflow absent from the original comparison. We distinguish the advance choice from best-of-three selection and a single execution from a repeated mean.

For ALFRED, the completed ten-task AgentSpec-OneShot evaluation scores 0.20, compared with 0.43 ± 0.12 for the selected evolved harness over three executions. These results quantify no-feedback component design in two environments and provide a direct reference for the corresponding iterated designs.

Scope of the control. AgentSpec-OneShot tests component design without execution feedback. Library access and contracts remain coupled, and its proposal budget differs from five-round evolution; the results therefore estimate workflow behavior rather than an isolated typing effect. A matched-budget Freeform-Evolve arm is outside the completed comparison.

AgentSpec's component-level contribution is tested directly by the ALFRED replacement. The round-3 and round-4 harnesses share reasoning, reflection, perception, and the execution loop. Only memory changes to align stored actions with their own outcomes. Their repeated means are 0.37 ± 0.12 → 0.43 ± 0.12. The trace identifies the defect, the contract provides the replacement boundary, and the surrounding harness remains fixed. This directly demonstrates AgentSpec's practical role in diagnosis, localized repair, and controlled component evaluation.

AuthorsOpenReview ↗

Response to Reviewer VSZz (2/4)

Q1: What is the maximum over fifteen uniform library draws on the development suite, reported for DeliveryBench, ALFRED and MiniGrid?

A: The fifteen-configuration DeliveryBench evaluation is complete. Configurations were sampled uniformly without replacement from the 90-configuration library, without a designer or execution-guided adaptation. Each configuration was evaluated on the complete development suite. We also report the full-suite configuration evaluations from the second deployment:

Environment / protocolEvaluated configurationsMaximumMean / median
DeliveryBench, sample set 1, temperature 0.11518.6111.16 / 12.81
DeliveryBench, sample set 2, temperature 1.0716.887.08 / 5.10
MiniGrid, complete ten-task suite100.500.32 / 0.30
ALFRED, complete ten-task suite60.700.35 / 0.35

The DeliveryBench maxima are ReAct+SimpleMem+Self-Refine at 0.1 and Direct+OpenClaw+Reflexion at 1.0. The MiniGrid maximum is attained by CoT+DynamicCheatsheet and Direct+Base+Self-Refine; the ALFRED maximum is CoT+Base+Reflexion.

The requested DeliveryBench maximum is 18.61. The MiniGrid and ALFRED rows report their actual full-suite sample sizes, rather than being labelled as fifteen-draw results. Only ten-task MiniGrid evaluations enter this comparison. The two DeliveryBench temperature settings remain separate.

The sample answers the library-selection question with a distribution of configurations rather than one hand-picked reference. We distinguish individual-configuration scores, selected maxima, and repeat means. The evolution campaign's 32.07 is a selected development score at temperature 1.0, so it is not a matched-temperature comparison with the fifteen-draw maximum at 0.1. The direct common-temperature repetitions are given in our response (3/4).


W2/Q2: Since the evaluation uses just two episodes from one map, could the authors clarify how reliable this improvement is?

A: We evaluated library configurations on five new DeliveryBench maps, with five episodes per configuration. Fourteen configurations have complete paired development and held-out results. The development winner, ReAct+SimpleMem+Self-Refine, is also the highest-scoring configuration among those fourteen on the held-out maps, at 30.91. Development and held-out scores have Pearson r = 0.65, p = 0.013; the held-out mean across configurations is 18.1.

Answer to the transfer question. Development scores contain transferable signal for these library configurations: the selected development winner also leads the evaluated five-map set, and scores correlate across suites. This analysis concerns library selection; no evolved-harness result is reported on these five maps, so 30.91 and the evolved development maximum of 32.07 are not a cross-harness comparison.

AuthorsOpenReview ↗

Response to Reviewer VSZz (3/4)

Q2: And since ALFRED carries the cost argument, why is it missing from the repeat table, when a three-repeat of the two 0.50 harnesses is cheap and would settle whether the tie, and the cost claim built on it, is real?

A: Yes, the repeated observations reproduce the central cost finding. We evaluated each frozen harness three times on the ten-task suite:

ALFRED harnessThree SR valuesMean ± sample SDCalls/episodeTokens/episode
Evolved, round-4 winner0.30 / 0.50 / 0.500.43 ± 0.1281.0452k
Earlier Freeform-OneShot0.40* / 0.40 / 0.500.43 ± 0.0612.128.8k
Evolved, round-3 winner0.30 / 0.50 / 0.300.37 ± 0.1295.0525k
Library ReAct+Base0.20 / 0.10 / 0.300.20 ± 0.1072.1425k

One task in the first one-shot execution encountered a simulator error before the agent acted. It remains unsuccessful in the ten-task denominator, yielding 4/10; among completed tasks that execution scores 4/9.

Answer to Q2. The selected evolved and one-shot harnesses have the same observed mean SR, 0.43, while the one-shot program uses approximately 6.7× fewer calls and 15.7× fewer tokens. For this frozen pair, the equal observed mean and measured cost reduction reproduce the performance–cost finding; this is an empirical comparison, not an equivalence test. The implementation analysis explains the computational difference: the one-shot program maintains more state in code and consults the executor less frequently.

The memory-only comparison also reproduces a positive mean change, 0.37 → 0.43. The original campaign values, 0.30 → 0.50, remain labelled as individual selected evaluations; the repeated mean difference is approximately 0.07.


W2/Q2: The DeliveryBench result seems somewhat unstable. The best score is 32.07, but the same harness only achieves 17.8 on average in repeated runs.

A: We added five complete-suite executions per frozen harness, fixing temperature 1.0 for all three:

DeliveryBench harnessFive complete-suite scoresMean ± sample SD
Evolved, round-3 winner18.91 / 6.08 / 17.24 / 20.92 / 14.8315.6 ± 5.8
Earlier Freeform-OneShot1.69 / 9.12 / 12.47 / 12.04 / 9.479.0 ± 4.3
Library CoT+MemoryBank22.01 / 22.77 / 14.31 / 25.31 / 25.9222.1 ± 4.6

The evolved harness's repeated mean exceeds the tested earlier freeform program's by 6.6 profit units. Two additional current-protocol freeform designs score 6.59 and −4.00 in single complete-suite executions. Thus the observed difference from the tested freeform designs is supported beyond the original campaign maximum.

Answer to the reliability question. The new common-temperature comparison measures a 6.6-unit mean advantage over the tested earlier freeform program. CoT+MemoryBank averages 22.1, so this evidence supports the evolved–freeform comparison, not the original claim of superiority over both baselines. The 32.07 selected campaign value is reported separately from the 15.6 frozen-harness mean. Each complete-suite score averages two seed episodes on one map; repetitions quantify execution variability on that suite, while the held-out library analysis addresses a separate transfer question.


Reviewer_WvjzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper appears to propose a stateful harness “engine” that manage harnesses, action map specifications, and representation for LLM-based agent in embodied environments. Using this framework, the authors compare three ways of constructing a harness for a fixed executor (Qwen3.5-9B) across five benchmarks: selecting a configuration from a human-written module library, having a coding agent (Claude Opus 5) write a standalone program in one shot without feedback, and having the coding agent iteratively revise reasoning, memory, and reflection modules over rounds of evaluation feedback.

Critical strengths

None

Critical weaknesses
  • The paper appears to be LLM-assisted without careful reviews and revisions by the authors
  • There are several jargons that are not explicitly defined before usage such as contracts, types, state lifecycles, execution boundaries, etc.
  • Despite “typed composition” being in the title, the paper never clearly defines what is a type, the justification for having a typed system, or the typing framework itself beyond superficial categorizations of tools and model capabilities.
  • There is no justification for (ambiguous) headlined design choices, such as “a complete harness is a value that can be serialized, diffed, and replayed”.
  • Since the reported comparisons are for stochastically constructed harnesses, the results appear to bear limited practical values in terms of generalizability and scalability.
  • There are also several unexplained and peculiar notations like J^\hat{J}, or “f02 and f03 share one protocol; f01 is an earlier version”.
  • The result tables are inconsistent
Questions for authors

The paper needs serious and careful revision for future submissions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Wvjz

We address each listed concern below, with the relevant submitted definitions and completed evaluations.


W1: The paper appears to be LLM-assisted without careful reviews and revisions by the authors

A: Page 10 discloses writing assistance and author responsibility. The accompanying responses provide additional experiments and reporting corrections. Disclosed assistance alone does not establish insufficient author review; please identify the specific passages supporting that judgment.


W2/W3: There are several jargons that are not explicitly defined before usage such as contracts, types, state lifecycles, execution boundaries, etc.

Despite “typed composition” being in the title, the paper never clearly defines what is a type, the justification for having a typed system, or the typing framework itself beyond superficial categorizations of tools and model capabilities.

A: Section 3, lines 154–169, and Tables 4–5 specify the runtime framework. A type is a declared input/output representation; a contract adds state persistence, resets, and action-outcome routing; a lifecycle defines when these updates occur; an execution boundary defines editable versus protected code. These specifications let generated components be replaced within a fixed embodied loop. The ALFRED memory-only repair demonstrates the practical value of that boundary. This is a runtime-interface specification, not a static type calculus.


W4: There is no justification for (ambiguous) headlined design choices, such as “a complete harness is a value that can be serialized, diffed, and replayed”.

A: The candidate JSON selects implementations and parameters; evaluated files are frozen. Serialization preserves a candidate, diffs identify its code changes, and re-execution measures the frozen candidate after selection. Table 2 uses this distinction. Re-execution under stochastic inference need not reproduce an identical trajectory.


W5: Since the reported comparisons are for stochastically constructed harnesses, the results appear to bear limited practical values in terms of generalizability and scalability.

A: Section 4 distinguishes development selection from frozen-harness evaluation. We add five DeliveryBench executions per harness, three ALFRED executions per harness, and five-map transfer tests for library configurations. The repeated ALFRED comparison yields equal observed mean SR of 0.43 with 6.7× and 15.7× differences in calls and tokens. These measured implementation and cost findings directly establish practical information beyond a single selected score.


W6: “f02 and f03 share one protocol; f01 is an earlier version”.

A: Section 5.1 and Appendix A.9 define f01/f02/f03 as separate programs and protocol versions; these responses use descriptive protocol names. The example symbol after “like” is absent from the review copy available to us; please identify it.


W7: The result tables are inconsistent

A: Our response to VSZz Q3 identifies the evaluation records behind the differing call counts, and the reporting clarifications supply the corrected entries and temperature settings. Please specify any additional table entries and the technical claim underlying the lowest soundness assessment.


Thanks to the authors for the rebuttal. Unfortunately, the revised manuscript has not addressed my fundamental concerns about the paper.

  • W1: My assessments are based on the substance of the paper itself, not on the disclosure alone. Your rebuttal responses and revised manuscript have not convinced me otherwise.
  • W2/W3: The revised Section 3 and Tables 4-5 only describe the components instead of providing concrete definitions.
  • W4: Your response and revision haven't explained why you would need these properties. Are they conceptually intrinsic to your proposed method? If so, how and why?
  • W5/W7: The exposition of experimental designs and presentation of results have not improved.
  • W6: This has not been addressed. f01, f02, f03 are now called draw 1, draw 2, draw 3, but the exposition of the results is still not clear. The "symbol" is in Equation 1.

Thus, I maintain my assessment and recommendation for substantial revision.

Thank you for the follow-up. Several concerns are already addressed explicitly in the revised manuscript; we clarify the relevant points below.

W2/W3 — Definitions. Section 3, lines 153–161, explicitly defines types, contracts, lifecycles, and execution boundaries. Table 4 specifies the corresponding inputs, outputs, and lifecycle operations. These are operational runtime-interface definitions. The statement that the revision provides no concrete definitions overlooks this text.

W4 — Purpose of the design properties. Serialization, implementation diffs, and re-execution support three specific operations: identifying the evaluated candidate, establishing the code change under intervention, and measuring execution variability of the frozen program after selection. Lines 84–86 and 188–196 explain these roles. They underpin the paper’s auditable evaluation procedure; we do not claim that every agent-design algorithm requires this implementation.

W6 — Notation. Equation 1 expresses selection among evaluated candidates: CBvalid\mathcal C_{\mathcal B}^{\mathrm{valid}} is the set of candidates validated and scored within budget B\mathcal B, and J^D(h;B)\widehat J_{\mathcal D}(h;B) is the measured development-suite score using executor BB. Making these definitions explicit beside the equation is a minor editorial clarification that changes neither the method nor the results. Appendix A.13, lines 1498–1505, separately explains the program identifiers and protocol differences.

W5/W7 — Experimental evidence. The revision adds common-temperature repetitions (Table 2), construction controls (Table 10), an observation-interface intervention (Table 13), and per-task and memory-audit evidence (Tables 16–17). The blanket statement that experimental design and presentation “have not improved” does not engage with these substantive additions.

Given the last-minute timing of this follow-up, we do not have sufficient time to make and verify another PDF revision before the discussion closes. We have therefore provided the relevant clarifications directly above. The remaining notation clarifications are minor editorial changes that do not affect the method or reported results.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers and Area Chairs (1/2)

We thank the reviewers for their evaluations. We appreciate Reviewer VSZz's recognition of AgentSpec's component traceability and performance–cost analysis, and Reviewer nCXQ's recognition of the explicit contracts, the localized ALFRED memory diagnosis, and the separation of design, evaluation, and deployment cost. These observations identify the central contribution: AgentSpec makes agent-written embodied harnesses composable, inspectable, and accountable at the component level.

To address the reviews, we added experiments in three areas: construction controls, repeated evaluation, and mechanism analysis. The key additions are summarized below; our individual responses provide the protocols and results.

ConcernAdditional evidenceMain finding
Attribution to feedback, library access, and contracts — VSZz W1/Q1AgentSpec-OneShot: component design with library access and contracts, without execution feedbackA single design pass produces executable designs scoring up to 15.81 on DeliveryBench; the controls quantify what composition achieves before feedback
Selection and execution variability — VSZz W2/Q2Five executions per DeliveryBench harness; three per ALFRED harnessSeparates selected campaign scores from repeated performance and directly tests the ALFRED cost comparison
Library selection — VSZz Q1Fifteen DeliveryBench library draws; full-suite samples on MiniGrid and ALFREDProvides the requested DeliveryBench maximum, 18.61, and a reference distribution beyond one fixed configuration
Practical value of component contracts — nCXQ W2/Q2ALFRED memory-only replacement, with three executions per harnessLocalizes an action–outcome alignment defect; repeated mean SR changes from 0.37 to 0.43 with the surrounding harness fixed
MiniGrid perception restriction — VSZz W1Three repeated evaluations per harness–suite cell under a structured-partial-view adapterQuantifies sensitivity to the observation interface and explains why editing scope must accompany workflow comparisons
Missing baselines and reporting — nCXQ W1/Q1; VSZz Q3; WvjzComplete ALFRED fixed-library evaluations, two new DeliveryBench freeform designs, call-count reconciliationFills the ALFRED and DeliveryBench comparisons and identifies the source of the apparent call-count discrepancies

The ALFRED repetitions reproduce a particularly informative systems finding: the selected evolved and one-shot harnesses both average 0.43 SR, while the one-shot program uses 6.7× fewer executor calls and 15.7× fewer tokens. This illustrates why harness evaluation should report how computation is divided between the model and code, alongside task performance.

These additions directly address the reviewers' requests for construction controls, repeated performance, missing baselines, and component-level evidence. AgentSpec contributes the runtime specification, state lifecycle, protected replacement boundaries, and implementation-linked execution records that make the analyses possible. We report completed evaluations and distinguish individual design scores, selected development maxima, and frozen-harness repeat means throughout.

AuthorsOpenReview ↗

General Response to All Reviewers and Area Chairs (2/2)

Evaluation Provenance and Reporting Clarifications

The reporting audit resolves the executor-call discrepancies and identifies the following corrections to the submitted text and figures:

Submitted itemCorrected reporting
MiniGrid one-shot calls “22–24”Approximately 22–38 across all draws; the earlier-protocol draw uses 37.5
One-shot programs generally finish in fewer environment stepsALFRED has 81–100 one-shot steps versus 69–82 across evolved round winners; executor calls and environment steps must be distinguished
ALFRED “seventeen-fold” token difference15.7-fold, from 523,392 / 33,324 for the recorded pair; the new repeated means also give approximately 15.7
Appendix A.3 “20 full-suite / 43 gate evaluations”30 full-suite / 45 gate evaluations in the audited four-campaign accounting
Appendix A.5 “42–55%” of one-shot tokens spent on failuresThese are success shares; failure shares are 45–91% across the reported draws
Appendix B.1/B.4 round-3 ALFRED goal-condition rate 0.420.37; 0.42 belongs to round 1
Figures 4(a)/7 earlier MiniGrid draw at 10/10The Table 1 replacement program scores 0.70; the earlier 10/10 plot describes a different program
Appendix A.4 DeliveryBench a/b/c orderingThe labels are misordered; candidate scores retain their recorded identities and values

The DeliveryBench campaign records place fourteen candidate evaluations at 1.0 and one unselected candidate at 0.1, correcting the submitted temperature description. All three harnesses in the new five-execution comparison use 1.0. Campaign maxima, frozen-harness repeat means, configuration-sample maxima, and held-out scores are reported separately.

The corrected repeated comparison is 15.6 ± 5.8 for evolution, 9.0 ± 4.3 for the earlier freeform program, and 22.1 ± 4.6 for CoT+MemoryBank. This updates the original DeliveryBench baseline-ordering statement. The runtime specification, localized state diagnosis, and repeated ALFRED performance–cost finding are supported by the corresponding implementation records and evaluations described above.

AuthorsOpenReview ↗

Guide to the revised manuscript to All Reviewers and Area Chairs

We have uploaded the revised manuscript. Reviewers found both the paper and, in places, our rebuttal hard to follow, so this note is a map: the final conclusions in one place, and a pointer to where each item sits in the revision. Nothing in the revision needs to be reconstructed by cross-reading the rebuttal threads.

Conclusions that changed under the new evidence

  1. DeliveryBench ordering. At the common executor temperature (1.0), over five complete-suite executions, the library configuration CoT+MemoryBank averages 22.1 ± 4.6, the evolved harness 15.6 ± 5.8, and the earlier freeform program 9.0 ± 4.3. The original claim that the evolved harness exceeds both baselines is withdrawn. The campaign value 32.07 is now labelled throughout as a selected best-of-5 maximum, not repeat performance.
  2. Temperature. The submitted campaign description was wrong: the campaign records place fourteen candidate evaluations at 1.0 and one unselected candidate at 0.1. The revision uses this corrected account in the setup text and the Table 2 caption, and reports the 0.1 library sample (15 draws, maximum 18.61) separately, never as a comparison against 1.0 results.
  3. MiniGrid. The gap is now attributed explicitly to the observation interface, not to the construction workflow alone: with a structured partial view, the frozen library harness improves on held-out tasks from 0.27 ± 0.12 to 0.77 ± 0.06, and the frozen evolved harness from 0.30 ± 0.20 to 0.63 ± 0.12, with implementations held fixed.

Conclusions that survived repeated evaluation

  1. ALFRED performance–cost. The selected evolved and one-shot harnesses have the same repeated mean success rate (0.43), while the one-shot program uses 6.7× fewer executor calls and 15.7× fewer tokens per episode.
  2. ALFRED memory repair. The memory-only replacement repeats at 0.37 → 0.43; we do not claim this difference is statistically established (one-sided p = 0.37). The supported claim is the mechanism: refused-action records mislabelled as "succeeded" fall from 292/365 (80%) in the original memory to 0/309 in the repaired one, with the surrounding harness unchanged.

Where each requested item is in the revision

RequestLocation in the revision
DeliveryBench claims recalibrated (VSZz)Abstract; Section 5.2; Section 6.3 — all restated on the common-temperature repeats above
Table 2 temperature description (VSZz)Setup text and Table 2 caption — one corrected, consistent account
MiniGrid interface implication (VSZz)Section 6.3 — stated explicitly as an interface implication
What the runtime checks / does not check (nCXQ)Runtime section and the ALFRED case — rejection checks, lifecycle dispatch, and unchecked semantics, with the claim narrowed to a traced, localized repair
Per-task memory-only outcomes (nCXQ)Full per-task table for the three repeats of each harness, with the executor-visible memory audit
All reporting correctionsIncorporated directly into the text and tables they correct — call counts, the 15.7× token figure, evaluation counts, and the temperature account — so the revision reads on its own, without the rebuttal threads

A ten-minute reading path through the revision: the Abstract, Section 5.2 with Table 2, and the final paragraph of Section 6.3 give every updated conclusion; Appendix A.8 and B.1 give the audit trail.

Thank you again for the scrutiny that produced these changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.