- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
AgentSpec presents a typed specification for embodied LLM-agent harnesses, with explicit interfaces and state lifecycles for perception, memory, reasoning, reflection, and action. In AgentSpec, a separate coding agent (Claude Opus 5 in CC) corrects and proposes changes to reasoning, memory, and reflection components using execution rollouts feedback. An executor model (Qwen 3.5 9B) remains fixed. The paper compares this iterative workflow with fixed human-written library configurations and standalone harnesses generated without execution feedback across DeliveryBench, ALFRED, MiniGrid, RoboTHOR, and DeliveryGym. Overall results vary by environment: the evolved harness outperforms the tested baselines on DeliveryBench. The Freeform baseline programs perform better on MiniGrid, including held-out tasks. The experiments clearly divide what each agent handle directly in code. The paper also carefully splits the cost for building the harness, testing it and running each agent (Proposer vs. Executor).
Critical strengths
- Explicit component contracts define each module’s inputs, outputs, and state lifecycle (Table 4).
- Concrete implementation analysis identifies an ALFRED memory bookkeeping error and documents its repair (Appendix B.1).
- The paper details cost reporting separates the resources used to develop, evaluate, and run each agent (Table 9).
Critical weaknesses
- Incomplete baseline comparisons. Table 1 lacks full Library-Fixed results for ALFRED and DeliveryGym, and Freeform-f02/f03 results for DeliveryBench. The omission of these full comparisons is not explicitly justified.
- Insufficient differentiation from prior work. AgentSquare already provides standardized modules and code evolution, while AFlow optimizes executable workflows using evaluation feedback. The additional practical benefits of AgentSpec’s runtime and lifecycle contracts need clearer justification.
- Difficult presentation. Many method names and run labels are introduced, while important experimental details are scattered across the main text and extensive appendices. This makes the setup difficult to reconstruct. The appendix section is too long to know what to read in priority.
Questions for authors
- Missing baselines: Could you provide the full Library-Fixed results for ALFRED and DeliveryGym, and the Freeform-f02/f03 results for DeliveryBench, under the same evaluation protocol? Why are these comparisons missing from Table 1?
- Contribution beyond prior work: Which runtime and state-management features distinguish AgentSpec from AgentSquare and AFlow? Could you demonstrate their practical benefits through a direct comparison or targeted ablations?
8 comments
Response to Reviewer nCXQ (2/2)
W2/Q2: Contribution beyond prior work: Which runtime and state-management features distinguish AgentSpec from AgentSquare and AFlow? Could you demonstrate their practical benefits through a direct comparison or targeted ablations?
A: The contribution is the embodied execution contract and the component-level analysis it enables. AgentSquare studies module evolution and recombination; AFlow studies executable workflow search. AgentSpec specifies how generated components participate in a persistent embodied loop: what they receive, what state survives each action, when state resets, and how executed-action outcomes return to memory and reflection.
The practical capabilities are:
| Runtime capability | Operational purpose | Evidence in this study |
|---|---|---|
| Declared interfaces and state lifecycles | Replace a component while preserving the surrounding loop | Table 4; ALFRED memory-only replacement |
| Explicit edit boundaries and frozen implementations | Identify which code change a candidate evaluation measures | Table 5; candidate specifications and implementation records |
| Component traces and cost accounting | Connect behavior to state updates and model-call patterns | ALFRED outcome diagnosis; repeated performance–cost comparison |
Targeted component evidence. Appendix B.1 identifies a recency memory that renders an action alongside the previous action's outcome. OutcomeTraceMemV1 uses the transition already supplied to memory to align each recorded action with its own consequence. The reasoner, reflection component, perception, and loop remain unchanged. The campaign scores are 0.30 → 0.50; three re-executions per harness yield 0.37 ± 0.12 → 0.43 ± 0.12.
Answer to the practical-benefit question. AgentSpec makes the state defect visible in a component trace, confines the repair to one replacement unit, and evaluates it with the surrounding harness unchanged. The repeated mean difference is approximately 0.07; the campaign's 0.20 difference is a single-evaluation observation. This demonstrates a concrete runtime capability: inspecting temporal state semantics and testing the corresponding localized repair.
The additional controls assess construction behavior. AgentSpec-OneShot produces a no-feedback DeliveryBench candidate scoring 15.81 and an ALFRED design scoring 0.20. The library samples quantify configuration selection without generated code. These are complementary controls for the studied workflows. Uniform library sampling is not a reproduction of AgentSquare's full algorithm, and the reported study does not include a direct AgentSquare/AFlow performance run.
W3: Difficult presentation. Many method names and run labels are introduced, while important experimental details are scattered across the main text and extensive appendices. This makes the setup difficult to reconstruct. The appendix section is too long to know what to read in priority.
A: The protocol can be read directly through the following distinctions:
| Workflow | Library access | Feedback during design | Editable implementation |
|---|---|---|---|
| Library-Fixed | Human-written components | None | Configuration only |
| Freeform-OneShot | No component-library access | None | Standalone harness, including observation processing |
| AgentSpec-OneShot | Yes | None | Reasoning, memory, reflection under contracts |
| AgentSpec-Evolve | Yes | Prior-round evaluations | The same component families, three candidates per round, five rounds |
The f01/f02/f03 names identify generated programs and protocol versions; the substantive distinctions are earlier-protocol versus current-protocol draws. Section 3 and Tables 4–5 specify contracts and editing scope; Sections 4–5 and Appendix A specify evaluation; Appendix B provides implementation cases. Appendix C is a separate component study with its own model and task settings. The definitions, compact protocol table, and new results in these responses make those distinctions explicit without requiring the reader to reconstruct them from run identifiers.
Thank you for the additional experiments and for being transparent about the corrections. These address most of my baseline concerns.
Reviewer VSZz raised concerns about the reliability of the DeliveryBench improvement. Tte library baseline CoT+MemoryBank now has a higher mean score than the evolved harness (22.1 vs 15.6), it changes one of the paper's central empirical conclusions . Together with the extensive corrections in the rebuttal, this makes it necessary to reassess which conclusions still hold, yet no revised manuscript has been shared and, without code, the numbers cannot be checked. I also agree with Reviewer Wvjz, the paper, and now the rebuttal, are very hard to follow.
I understand that the intended contribution centres on the embodied harness specification and its component analysis. Before finalising my assessment, two clarifications would help:
-
In the ALFRED example, the buggy memory satisfied its contract, so the contracts did not catch the bug. What did they add in this example compared with ordinary modular code that logs every step? And what exactly does the runtime check or enforce?
-
Could you provide the per-task outcomes across the three repeated runs of each harness in the memory-only comparison, and explain whether the improvements consistently correspond to the action–outcome alignment defect?
These clarifications would help me judge whether the new data establishes a distinctive and supported runtime contribution.
Response to Reviewer nCXQ (1/2)
We thank the reviewer for recognizing the explicit component contracts, the memory diagnosis, and the cost accounting. We added the complete ALFRED library evaluation and the two missing DeliveryBench freeform designs, and clarify the provenance of the empty cells below.
A: The gate explains the original missing library cells. In a gated campaign, all three candidates receive gate evaluations, and only the gate winner proceeds to the complete suite. ReAct+Base scored 0.00 on ALFRED's four-task gate and was not selected for full-suite evaluation. DeliveryGym likewise had a gate-only library record. An empty cell denotes a full-suite evaluation that was not performed, rather than a zero score or an omitted completed result.
ALFRED Library-Fixed is now evaluated on the full suite. Three ten-task executions yield 0.20, 0.10, 0.30, or 0.20 ± 0.10. The selected evolved and earlier freeform harnesses average 0.43 ± 0.12 and 0.43 ± 0.06, respectively. This completes the particular fixed-reference comparison and directly addresses the ALFRED request.
The broader library sample is reported separately: among six complete ALFRED configuration evaluations, CoT+Base+Reflexion reaches 0.70 in one execution. This distinguishes performance against the fixed ReAct+Base reference from performance against a searched library maximum. Our response to VSZz gives the sample sizes and distributions for all three environments.
DeliveryBench now includes the two additional current-protocol freeform designs:
Both new programs were generated in fresh no-feedback sessions and evaluated on both episodes. The −4.00 program completed evaluation without completing a delivery. The sessions use the study's designer model and effort through an unattended API route with a spending cap; this deployment difference is stated alongside the results. The evolved harness's repeated mean is 15.6 ± 5.8; the full three-harness repetition table, including CoT+MemoryBank, appears in the response to VSZz Q2.
DeliveryGym coverage: ReAct+Base has a gate-only score of 8.33 on four shifts. The full 32-shift library score is unmeasured and labelled accordingly; the gate score is not used in the full-suite comparison.