acceptodds
Under review as a conference paper at ICLR 2027

ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents

Abstract

Interactive agent benchmarks face a tension between scalable construction and realistic workflow evaluation. Hand-authored tasks are expensive to extend and revise, while static prompts and clean-state environments miss failures that arise only when agents operate over persistent, partially completed workflows. Existing interactive benchmarks have advanced agent evaluation significantly, but do not systematically test whether agents preserve valid artifacts, repair stale state, replace incorrect objects, or reconcile conflicting evidence before acting. We present ClawForge, a generator-backed benchmark framework for executable command-line workflows under state conflict. The framework compiles scenario templates, grounded slots, initialized state, reference trajectories, and validators into reproducible task specifications, then evaluates agents step by step using normalized end state and observable side effects rather than exact trajectory matching. We instantiate the framework as ClawForge-Bench, a 362-task snapshot spanning 17 scenarios and 6 ability categories. Results across 11 model endpoints show that strict full-pass accuracy ranges from 35.9% to 61.6%, Wrong-State Replacement ranges from 0.0% to 52.2%, and Interrupted Workflow Resume ranges from 16.7% to 100.0%, revealing strongly non-nested capability profiles. Partial-credit and step-efficiency analyses further show that many failures are near-miss closures rather than early breakdowns, and that models exhibit qualitatively different failure styles under state conflict.

Primary area:
datasets and benchmarks
Keywords:
Command-Line Agents, Agent Evaluation, Interactive Environments, Generation Harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9w82OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

FlawForge benchmark is proposed to evaluate agents under different workflow state, where tasks may already be partially completed, inconsistent, stale, or duplicate-sensitive before task execution begins. It finds that model performance differ in tasks such as “wrong state replacement” and “interrupted workflow resume”.

Critical strengths
  1. It is quite nice to have partial credits besides final success for long-running agentic jobs.
  2. The snapshot selection setup is quite nice to have.
Critical weaknesses
  1. Clarity. I have no clue what an actual task instance would look like until I look into the appendix. Even the appendix and the demo episode in Figure 1 (right) are too marked up to illustrate a concrete task.
  2. 17 task templates really is quite limited for a dataset.
  3. The task selection and primary ability taxonomy seem to have appeared out of no where. It does not make sense to me why “information transfer” is related to state repair and why it is part of the 6 primary abilities. Please also see questions below.
Questions for authors
  1. How are the 17 base task templates created? Why are they useful, representative, different from each other, and worth testing?
  2. How did you create the 6-ability taxonomy? Why are they important to measure? Do they correlated in what a solution demands? Also according to Table 2, one of the abilities are already quite saturated (duplicate avoidance).
3 comments

We thank the reviewer for highlighting the usefulness of partial credit for long-running agentic workflows and the value of the snapshot-selection setup. We appreciate your recognition of these components, which support our goal of examining workflow progress under a fixed, inspectable task configuration. Below, we respond to each of your comments and questions point by point.

W1: What does an actual task look like, and can it be explained clearly before the diagrams and appendix?

Response: We move a plain-language workflow example to the first paragraph of the introduction. It states the user request, existing objects, intended changes, and preservation requirement directly, before the diagrams.

The example asks the agent to finish a release handoff and send a recap while preserving an existing next-step task and calendar sync. The required behavior is to inspect, write the handoff, and send the recap without recreating those objects. The corresponding concrete task is hard_decision_workflow_307 in §4.3. We also distinguish its original misplaced review-note seed from the corrected candidate rather than implying the original file was accessible.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Introduction, first paragraph; §4.3; Appendix G.1. This is a presentation correction, not a new model-performance experiment.

W2: Does a dataset built from only 17 templates provide sufficient structural coverage?

Response: We agree that 362 instantiated tasks do not amount to 362 independent workflow designs. We now state controlled coverage explicitly and do not claim broad representativeness or offer extra city/recipient substitutions as new structural diversity.

The evaluated release remains 17 scenarios and 362 tasks. The revision adds a coverage matrix, scenario-equal aggregation, and scenario-cluster sensitivity analyses; it does not add independent templates. Adjacent micro rankings are unresolved in the 17-cluster analysis—for example, Astra minus Sol is 3.9 percentage points with a 95% interval of [−8.7, 14.2].

We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1, §4.2.1, and Appendix G.4–G.5. Broader template coverage remains untested.

W3: Why were these tasks and six labels selected, and how does information transfer relate to state repair?

Response: We distinguish implementation, design rationale, and historical provenance. The released generator defines scenario builders and assigns primary/overlapping labels. These labels organize workflow diagnostics; information transfer is not asserted to be a subtype of state repair. The original procedure that selected the templates and six labels still requires author confirmation.

We recover the full 17-scenario mapping, six primary labels, and overlapping tags. Three primary information-transfer scenarios contain 32 tasks; they also carry workflow-completion tags. The scenario-tag matrix has no family tagged with both information transfer and state repair. This describes the implemented labeling, not empirical independence between cognitive abilities.

We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1 and Appendix G.5. Q1a and Q2a below explicitly identify the remaining provenance gaps.

Question 1: Creation, utility, representativeness, distinctness, and value of the templates

Q1a: How were the 17 base task templates created?

Response: For reproducible construction, the frozen hard-workflow generator contains a builder for each scenario that specifies base state and overrides, instruction renderings, reference commands, and checks, then instantiates grounded slots. The framework's generation-time LLM pipeline for PRO scenario material is a separate code path. This distinction does not establish the upstream authorship or selection history of the original hard templates.

All 362 frozen task specifications match the recorded SHA-256 fingerprint and 17-scenario mapping. Reference replay yields 355 evaluator passes and exactly seven known failures. Historical records identifying who designed the initial templates, which LLMs assisted which steps, or how candidates were selected have not been verified. That part of the question remains pending author confirmation; no unsupported crowdsourcing or expert-design protocol is claimed.

We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1; Appendix G.1/G.5; the version manifest and task-level construction inventory.

Q1b: What makes these templates useful for evaluating agents?

Response: Their intended diagnostic use is to distinguish actions that superficially look productive from actions appropriate to the current state: preserving an existing object, filling a gap, or retiring stale state before closure. This is an operational motivation, not a measured claim of improved real-world productivity.

The recorded check-level analysis identifies concrete incomplete workflows. For GPT-5.5's 25 task-type Replacement failures, calendar creation fails in 25/25, task creation in 22/25, and stale-task retirement in 18/25, with overlapping counts. The audit also exposes scoring defects, so these observations are presented as versioned operational diagnostics rather than validated latent-ability measurements.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Plain task example in the introduction; §4.3; Appendix G.5.

Q1c: What evidence supports the templates' representativeness?

Response: We do not have an empirical basis for claiming that 17 scenarios are a statistically representative sample of real assistant work. We revise the scope to controlled coverage of specified workflow conditions.

The coverage matrix records all 17 scenarios' base profiles, decision tags, and checked surfaces, and the task inventory records both prompt styles. It documents what is covered and where templates overlap; it is not a representativeness study. No user-workflow corpus, frequency-weighted sampling study, or external-domain validation has been added.

We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1 and Appendix G.5, scenario coverage table. Broad representativeness remains unsupported and is not claimed.

Q1d: How do the templates differ in starting states, decisions, and required outcomes?

Response: We compare differences in initial state, intended decisions, output surfaces, and preservation/repair obligations rather than relying on scenario names or slot substitutions. Some templates deliberately overlap, which we now make explicit.

The audit tests 38 scenario/base-state/check-signature combinations covering all 17 scenarios. The repair pair shares model/calendar/task base-state variants, while State Repair additionally requests a handoff and has a different subtype distribution: 13/16/17 versus Replacement's 8/13/25. Merging them into one 92-task family gives 58 Astra, 43 Sol, 33 GPT-5.5, and 19 Luna passes. Neither 38 tested combinations nor two template names is treated as proof of independent abilities.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Appendix G.3–G.5; scenario coverage and repair-subtype tables.

Q1e: Why are these workflow tasks worth testing?

Response: The proposed value is controlled inspection of whether an agent leaves the intended state behind, including preservation and downstream effects that command-text matching would not capture. We separate that motivation from claims about semantic validity and deployment usefulness.

The executable setup permits concrete, reproducible audits: all 1,448 stored GPT pass/fail outcomes reproduce, while reference replay identifies seven inconsistent reference/check pairs and 17 passing traces with failed note reads. In the error-free stress subset, the original evaluator accepts 82/136 designated negatives. These results show both the auditability of the setup and the need to strengthen its checks; they do not establish external utility or broad agent reliability.

We have incorporated the discussion and evidence into the revised manuscript: §4.2.5 and Appendix G.1–G.3.

Question 2: Creation, importance, overlap, and saturation of the ability taxonomy

Q2a: How was the six-ability taxonomy chosen?

Response: The implementation assigns each scenario one primary diagnostic label and additional overlapping tags. We describe that operational assignment and the definitions used in the release. We do not claim a learned taxonomy, a validated latent-factor model, or an expert-consensus study.

Six primary labels partition all 362 tasks, and the scenario/tag mapping is recovered for every family. The historical process that originally chose those six labels has not been verified and still requires author confirmation. Recovering a code mapping does not answer that historical question by itself.

We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1; Appendix G.5 and the primary-label/tag inventory. Design-history confirmation remains pending.

Q2b: Why are these six abilities important to measure?

Response: Each label describes a different operational question that a stateful workflow can pose. The purpose is diagnostic organization, while a single workflow may require several of them.

Primary labelQuestion it organizesTasks
Duplicate avoidanceShould existing valid work be preserved instead of recreated?20
Gap completionWhich missing part should be added while retaining completed work?52
Information transferIs information carried into the required artifact or communication?32
Multi-source reasoningWhich branch follows from evidence across sources?86
State repairWhich stale configuration or object needs correction?92
Workflow completionAre the required downstream operational effects produced?80

The counts describe the release, not the population importance or independence of these capabilities. Information transfer and repair are separate primary labels; both can serve the broader goal of workflow completion.

We have incorporated the corresponding discussion and evidence into the revised manuscript: §2.1; Appendix A.1 and G.5.

Q2c: Do the ability categories overlap or correlate in what a solution demands?

Response: The recorded tags explicitly overlap. We add scenario-level co-occurrence counts to document joint demands, counting a template once rather than treating its slot variants as independent evidence. These descriptive counts are not behavioral correlations between latent abilities.

Workflow completion is tagged in 14/17 scenario families. It co-occurs with information transfer in three families, gap completion in three, multi-source reasoning in two, state repair in two, and duplicate avoidance in two. Gap completion and duplicate avoidance co-occur in two families. No independent psychometric correlation or latent-factor analysis has been performed.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Appendix G.5 and the scenario-level tag co-occurrence matrix. We make no claim that the six labels are statistically independent.

Q2d: Why retain duplicate-avoidance tasks when that category is already near saturation?

Response: We retain these tasks as basic preservation controls, not as a principal source of frontier-model separation. A benchmark of stateful behavior should make successful restraint visible alongside repair and completion failures.

The primary duplicate-avoidance slice contains 20/362 tasks (5.5%), spanning two scenario families. Recorded accuracy on this slice is 90–100% across the nine GPT/Kimi endpoints with checked-in task-level data. The original cross-family table also shows near-saturation. We state that ceiling behavior explicitly and use scenario-cluster/equal-weight analyses so repeated easy variants are not mistaken for independent breadth.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Primary-label results table; §2.1 and Appendix G.4–G.5. No additional hard duplicate-avoidance scenarios or reruns are claimed.

Reviewer_bNKAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ClawForge generates executable command-line tasks with initialized state, reference trajectories, and validators. Its benchmark contains 362 tasks across 17 scenarios, testing whether agents preserve, repair, or replace existing state. Evaluation of 11 model endpoints shows varied strengths across scenarios, with strict accuracy ranging from 35.9pp to 61.6pp.

Critical strengths

(1) The benchmark targets useful workflow behaviors, including avoiding duplicates, repairing stale artifacts, and completing interrupted work. Coupling tasks with initial states and executable checks supports controlled evaluation. (2) Scenario-level results reveal differences hidden by overall accuracy. Joint reporting of strict success, partial credit, and executed steps provides useful diagnostic evidence.

Critical weaknesses

(1) Evaluation validity needs stronger support. In Appendix F, Example E requires a specific model switch that is not stated in the displayed instruction; its source in agent-visible information is unclear. Table 6 also shows count and filename checks without demonstrating that they reject semantically incorrect artifacts. (2) Cross-endpoint comparisons use different step budgets and serving paths. The four newer GPT endpoints are better controlled, but their bootstrap intervals measure task-sampling uncertainty rather than repeated-run variation. This limits confidence in small ranking differences and model-specific failure patterns.

Questions for authors

(1) Where can the agent discover the required model in Example E, and are all evaluated requirements available through instructions or observations? (2) Have the validators been tested on incorrect file contents, unrelated calendar events, and valid alternative solutions? What are their error rates? (3) Do the main rankings and contrasting scenario profiles persist across repeated runs with matched interaction budgets?

3 comments

We thank the reviewer for recognizing the value of evaluating duplicate avoidance, stale-state repair, and interrupted-workflow completion, as well as the controlled evaluation supported by coupling tasks with initialized states and executable checks. We also appreciate your acknowledgment that scenario-level profiles and joint reporting of strict success, partial credit, and executed steps provide diagnostic evidence beyond aggregate accuracy. Below, we respond to each of your comments and questions point by point.

W1: Are required targets available to the agent, and do validators reject semantically incorrect artifacts?

Response: We investigate target discoverability and validator specificity separately. The frozen instruction does not identify Example E's desired model, and the stress tests confirm that artifact existence alone can accept incorrect outputs. We narrow the interpretation of the original scores and provide a versioned, targeted correction.

The 2,046-check inventory flags 42 desired-model requirements for which a target policy was not identified, including 25 Release Gate tasks. On probes with error-free execution, the original evaluator accepts 82/136 designated negatives and all 102 intended-positive controls. The city-binding correction reduces negative accepts to 51/136 while retaining those 102 controls. It does not validate file meaning or resolve all hidden targets.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main text §4.2.5; Appendix G.2–G.3. Individual tests and rates are answered separately in Q2a–Q2d.

W2: Different budgets, serving paths, and unmeasured run-to-run variation limit confidence in rankings and scenario profiles.

Response: We focus the controlled sensitivity analyses on the four newer GPT endpoints sharing the frozen tasks and a 20-step cap. Astra's gateway revision remains different and is explicitly documented. We replace adjacent-ranking confidence claims with scenario-cluster sensitivity results and acknowledge that no independent repeated-run variance was measured.

Astra-minus-Sol's scenario-cluster interval is [−8.7, 14.2] percentage points, and Sol-minus-GPT-5.5's is [−2.0, 15.8]. Both include zero. Replay reproduces all 1,448 historical pass/fail outcomes, but supplies zero new independent model runs. It therefore verifies scoring reproducibility only.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main text §4.2.1 and §4.2.5; Appendix D.1 and G.1/G.4.

Q1a: Where can the agent discover Example E's required model?

Response: We have not identified a desired-model policy in the inspected frozen instruction or seeded evidence. The current model and the catalog of available models do not specify which model is required. We acknowledge the mismatch instead of describing it as reasonable hidden difficulty.

Example E checks a specific model despite its instruction asking only to confirm the current model. More broadly, 25 of the 30 Release Gate tasks receive a target-policy evidence flag. Separately, seven Release Gate references fail because their commands omit an evaluator-required model transition. A policy flag and a failing reference are distinct findings.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Appendix G.2. This remains a specification issue requiring author adjudication, not a completed semantic fix.

Q1b: Are all scored requirements available through instructions or observations?

Response: We cannot substantiate that blanket claim for the frozen release. We now inventory every check and preserve per-task instruction, catalog, benchmark notes, current configuration, and seeded material. Automated evidence categories identify open cases; they do not replace a human semantic audit.

Of 71 model-setting checks, 10 name the target in the instruction, 19 start in the evaluator-required model without an identified desired-model policy, and 42 have an unidentified target policy. The inventory covers all 2,046 checks over 362 tasks. We also find 17 passing references with failed review-note reads caused by a seed path outside the materialized workspace. Corrected seed paths eliminate those command errors, but do not establish discoverability of every requirement.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main text §4.2.5; Appendix G.1–G.2. Human adjudication of flagged requirements remains pending.

Q2a: Can the validators reject files with incorrect contents?

Response: Yes. We replace reference file contents with an unrelated grocery list while retaining the file path and other commands. We separately exclude command-error traces when reporting the controlled probe counts.

The original evaluator accepts 10/10 error-free wrong-content probes. The city-binding correction also accepts 10/10, because it binds task/event titles rather than validating file meaning. This test exposes a remaining limitation; we do not claim it has been fixed by the targeted correction.

We have revised Appendix G.3; concrete command-level probes are preserved in the audit artifact.

Q2b: Can the validators reject unrelated calendar events?

Response: Yes. We replace the required event title with an unrelated dentist appointment, preserving other commands.

Among error-free probes, the original evaluator accepts 16/17 unrelated-event cases (94.1%). Requiring an event title to refer to the city explicitly named in the instruction reduces acceptance to 1/17 (5.9%). This narrower object-binding check does not establish full calendar-event semantics or correct timing.

We have incorporated Appendix G.3 and the versioned candidate specifications.

Q2c: Do the validators accept valid alternative solutions?

Response: Yes, within a deliberately narrow control family: reordering independent read-only inspections or adding a harmless inspection. We do not present these as exhaustive coverage of valid workflow alternatives.

The error-free positive-control set contains 34 references and 68 inspection alternatives. All 102 are accepted by both the original and city-binding evaluators: zero rejections in this constructed set. The complete 114 intended-positive controls include 12 traces with command errors, which are separately flagged and not used as the clean positive denominator.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Appendix G.3.

Q2d: What are the validators' measured error rates?

Response: We report exact constructed-probe acceptance/rejection counts, with denominators. These measure performance on the designed stress suite, not error prevalence in natural model rollouts. Independent human-scored semantic rates are not yet available.

On probes with error-free execution:

Designated-negative classNOriginal acceptsCity-binding accepts
Incorrect file contents101010
Unrelated calendar event17161
Wrong task object19171
Omitted required mutation3400
Duplicate created object272727
Wrong time or recipient291212
Total13682 (60.3%)51 (37.5%)

Both evaluators reject 0/102 error-free intended-positive controls. The broader diagnostic suite accepts 95/154 negatives, but includes 18 negative traces with command errors and is reported separately. Eighty-five blinded historical-trajectory packets and two real-author forms are prepared; completed human annotations currently number zero, so no human agreement or semantic false-positive/negative rate is claimed.

We have revised Appendix G.3 and G.6. The remaining content, duplicate, and timing failures are disclosed.

Q3a: Do the main rankings persist across independent runs with matched interaction budgets?

Response: We have not established this: no new independent model runs are available. Common-budget analysis of existing runs and deterministic action replay cannot replace repeated model evaluation.

The four newer GPT records all use a 20-step cap. Their original pass counts are 223/362, 209/362, 185/362, and 148/362 for Astra, Sol, GPT-5.5, and Luna. Removing the entire Release Gate family gives 65.7%, 61.4%, 54.2%, and 43.1%. Replaying stored actions with targeted city checks gives 217/362, 195/362, 168/362, and 134/362. These preserve the observed ordering under the stated offline changes; they do not show ordering stability across independent runs.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main-text sensitivity table; Appendix G.1/G.4. Astra's serving revision remains a comparison limitation.

Q3b: Do contrasting scenario profiles persist across independent, budget-matched runs?

Response: This is also not established without independent reruns. We instead decompose the existing contrasts by subtype and recorded failed checks, and avoid describing a one-run difference as a stable latent capability.

GPT-5.5 records 16/17 task-type State Repair passes versus 0/25 task-type Replacement passes. All 25 Replacement failures lack the calendar-creation check; task creation fails in 22/25 and retirement in 18/25, with overlapping counts. These operational contrasts appear in the saved records, but no across-run distribution for them has been measured.

We have revised the main text §4.3; Appendix G.5.

Thank you for the detailed rebuttal and additional results. I will keep my current score.

Reviewer_VxgBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ClawForge is a benchmark for command line agents where tasks start from preexisting, partially completed state instead of a clean environment. Agents have to inspect the task board, calendar, email, files and cron jobs, then keep what’s valid, fix or replace what’s stale, and add only what’s missing without creating duplicates. Tasks are generated from 17 scenario templates by filling in slots like city, timezone and recipient, and each task bundles its initial state, a reference solution and executable checks. Grading is done on the final state and side effects rather than matching exact commands, with both strict pass and partial credit.

Critical strengths
  1. Testing agents on messy existing state: Starting tasks from partially done, stale or conflicting state is a really useful idea. This is what real assistant workflows look like, and it catches failures like duplicating existing work or leaving stale state behind that clean-state benchmarks miss entirely. 2Clear grading and concrete examples: Checking the final state instead of exact commands allows multiple valid solutions, and partial credit shows that many failures are near misses where the agent skips one last step. The case studies in Section 4.3 and the full scenario catalog in Appendix F make the benchmark easy to understand, since each scenario comes with a real task instance, its starting state, a sample trace and what the checker looks for. This level of detail also made it possible to spot issues like the Release Gate overlap, which says a lot about how transparent the setup is
Critical weaknesses

1.Statistical independence: The 362 tasks come from 17 templates with only slot changes like city or recipient, so tasks within a scenario are close to copies. The bootstrap intervals in D.3 resample individual tasks, which likely understates the real uncertainty, especially for close comparisons like Astra vs Sol. 2. Overlapping scenarios: State Repair and Wrong-State Replacement look nearly identical from Examples K and O, where both require retiring a stale task and creating a replacement task and sync. Yet GPT-5.5 scores 71.7% on one and 0% on the other. Without knowing what separates them, it’s hard to tell if this gap reflects a real ability difference or a template artifact, and this gap is a big part of the paper’s non-nested profiles argument. 3. Release Gate validity: Based on the catalog IDs and scenario counts, Release Gate seems to cover tasks 43 to 72, which would put all seven tasks with broken references (D.1) inside it. That would make about a quarter of the scenario unsolvable as specified, while the paper presents it as uniformly hard and uses it in the shared-failure analysis (Section 4.2.4). Example E also requires switching to a model the instruction never names.

Questions for authors

A. Clarifications

  1. Release Gate IDs: Can you confirm whether the seven audited tasks are all Release Gate tasks?
  2. Required model: In Example E the instruction says to confirm the current model but doesn’t name the required one, which the paper calls a hidden constraint. How is the agent expected to know which model to switch to?
  3. State Repair vs Wrong-State Replacement: What exactly differs between these two templates?

B. Additional analysis (can be done on existing runs)

  1. Clustered intervals: Could you report bootstrap intervals resampled by scenario?
  2. Release Gate rescoring: How do Release Gate results and the shared-failure analysis (Section 4.2,4) look with the 7 broken tasks removed?
3 comments

We thank the reviewer for recognizing the practical value of testing agents on partially completed, stale, and conflicting workflow states. We also appreciate your positive assessment of the transparent grading setup, concrete scenario examples, and the diagnostic value of partial credit, which make the benchmark's behavior and evaluation conditions easier to inspect. Below, we respond to each of your comments and questions point by point.

W1: Tasks share templates, so task-level bootstrap may understate uncertainty in model comparisons.

Response: We agree that resampling individual tasks does not address template dependence. We add paired scenario-cluster bootstrap, retaining every task in each sampled scenario and using identical sampled clusters for both models: 100,000 draws, seed 42. With only 17 scenarios, this is a clustering sensitivity analysis, not an assumption that the scenarios are an IID sample of all workflows.

Astra minus Sol remains 3.9 percentage points, but its 95% interval widens from the original task-bootstrap [−0.8, 8.6] to the scenario-cluster [−8.7, 14.2]. The corresponding Sol-minus-GPT-5.5 interval is [−2.0, 15.8]. We no longer describe these adjacent rankings as resolved under clustered analysis.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main text §4.2.1; Appendix G.4. Full pairwise results are given separately in B1 below.

W2: State Repair and Wrong-State Replacement appear nearly identical, yet GPT-5.5 performs very differently.

Response: These templates overlap structurally and are not independent latent abilities. Both contain model, calendar, and task variants; State Repair additionally requests a handoff. Wording, surface titles, and subtype proportions also differ. We therefore report subtypes and recorded check failures, rather than attributing the gap to replacement reasoning alone.

GPT-5.5's 33/46 versus 0/46 aggregate contrast decomposes as follows:

SubtypeState Repair passesReplacement passes
Model3/130/8
Calendar14/160/13
Task16/170/25

Among the 25 task-type Replacement failures, calendar creation fails in 25/25, task creation in 22/25, and retirement in 18/25; these counts overlap. Those three checks pass in all 17 task-type State Repair instances. This localizes recorded failure checks, but does not isolate a causal ability difference or exclude a wording effect.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main text §4.3; Appendix G.5 and the subtype/check-level audit tables.

W3: Release Gate may appear artificially difficult because of inconsistent references and an unstated model requirement.

Response: We separate reference consistency from target discoverability. The seven inconsistent references do all belong to Release Gate. Removing them addresses that known inconsistency, but does not establish that all remaining targets are agent-discoverable. We revise the shared-failure interpretation accordingly.

All seven reference failures reproduce. Removing them changes the four newer GPT endpoints' Release Gate result from 5/30 to 5/23 (21.7%). Astra and Sol's joint failures fall from 109 to 102; Branch Resolution plus Release Gate account for 44/102 (43.1%), rather than 51/109 (46.8%). Separately, the requirement inventory flags 25 Release Gate model targets for which a desired-model policy was not identified. These are evidence flags requiring semantic adjudication, not a certified count of unsolvable tasks.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main text §§4.2.4–4.2.5; Appendix G.1–G.3. A2 addresses the unstated model directly; B2 gives the full rescoring results.

A. Clarifications

A1: Are all seven audited reference failures Release Gate tasks?

Response: Yes. The frozen task metadata identifies each of the seven as Release Gate.

Release Gate comprises IDs 43–72, totaling 30 tasks. The seven exceptions are 44, 48, 49, 57, 64, 68, and 69: 7/30, or 23.3% of the scenario. Reference replay reproduces exactly these seven evaluator failures, with 355/362 references satisfying their check sets.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Appendix G.2; the task-level reference-replay artifact preserves IDs, failed checks, and command errors.

A2: What agent-visible evidence tells the agent which model to select in Example E?

Response: In the inspected frozen task, we have not identified an agent-visible policy specifying the desired model. Asking the agent to confirm its current model, or presenting available models in the catalog, does not identify a required replacement. We acknowledge this as a specification concern rather than defend it as legitimate hidden difficulty.

Across all 2,046 checks, 71 check the model setting: 10 targets are explicit in the instruction, 19 are already satisfied by the initial configuration without an identified desired-model policy, and 42 require a target policy that the inventory did not identify. The 42 flagged cases comprise 25 Release Gate, 10 State Repair, and seven Replacement tasks. This automated inventory is not a completed human semantic certification; additional target requirements remain unresolved.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main text §4.2.5; Appendix G.2. The revision distinguishes discoverability from a reference's ability to satisfy the evaluator.

A3: What exactly differs between State Repair and Wrong-State Replacement?

Response: Both select from the same three base-state types. Their differences include the handoff requirement, instruction wording, artifact titles, and the sampled subtype composition.

PropertyState RepairWrong-State Replacement
Model/calendar/task counts13/16/178/13/25
Handoff request/checksPresentAbsent
Task naming in referencesRelease follow-throughReplacement next step
Task/calendar/model remediation structureShared base patternsShared base patterns

Merging them into one 92-task repair family gives 58/92 passes for Astra, 43/92 for Sol, 33/92 for GPT-5.5, and 19/92 for Luna. When also treated as one bootstrap cluster, Astra-minus-Sol's micro interval is [−10.4, 14.0] percentage points. We do not claim an experiment that independently varies wording while holding every other factor fixed.

We have incorporated the corresponding discussion and evidence into the revised manuscript: Appendix G.4–G.5; the coverage matrix and merged-family bootstrap artifact.

B. Additional analysis

B1: Can uncertainty intervals be recomputed by resampling entire scenarios?

Response: Yes. We report paired cluster percentile intervals using the 17 scenarios as resampling units. Micro accuracy divides summed passes by summed task counts; macro accuracy gives each scenario equal weight.

All differences and intervals below are in percentage points.

ComparisonMicro difference [95% interval]Macro difference [95% interval]
Astra − Sol3.9 [−8.7, 14.2]3.1 [−6.0, 11.3]
Sol − GPT-5.56.6 [−2.0, 15.8]4.7 [−2.7, 12.5]
GPT-5.5 − Luna10.2 [−0.3, 22.0]9.9 [2.5, 18.0]
Astra − Luna20.7 [7.4, 32.6]17.7 [7.9, 27.5]

We also provide merged-repair clustering and leave-one-scenario-out sensitivities. None estimates decoding, serving, or independent repeated-run variance.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main text §4.2.1; Appendix G.4. The artifact contains 24 cluster comparisons and 200 leave-one-scenario-out comparisons.

B2: How do Release Gate scores and shared failures change after removing the seven inconsistent references?

Response: We apply the same seven-ID exclusion to every model and retain the original results alongside it. We also exclude the entire Release Gate family as a separate sensitivity check.

ModelOriginal 362Exclude seven: 355Exclude all gate: 332
Astra223/362 = 61.6%223/355 = 62.8%218/332 = 65.7%
Sol209/362 = 57.7%209/355 = 58.9%204/332 = 61.4%
GPT-5.5185/362 = 51.1%185/355 = 52.1%180/332 = 54.2%
Luna148/362 = 40.9%148/355 = 41.7%143/332 = 43.1%

Within Release Gate, all four pass 5/23 after exclusion, with mean partial credit 0.8696. For Astra and Sol, shared failures change as follows:

QuantityOriginalExclude seven
Joint failures109102
Joint failures in Branch Resolution + Release Gate51/109 = 46.8%44/102 = 43.1%
Tasks in those two scenarios70/362 = 19.3%63/355 = 17.7%

The observed ordering persists under these exclusions, but does not certify semantic validity or repeated-run stability. We also disclose 17 passing reference traces with failed review-note reads; a separately versioned seed-path correction eliminates those command errors without removing the seven model-target evaluator failures.

We have incorporated the corresponding discussion and evidence into the revised manuscript: the main-text sensitivity table and §4.2.5; Appendix G.1–G.2.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback and recognition of the workflow setting and diagnostic evaluation. Below, we summarize the main revisions; our point-by-point replies address each concern separately.

  1. Added a concrete workflow example explaining existing state, required changes, and preservation obligations. (Introduction; §4.3. Reviewer 9w82.)

  2. Clarified the construction of the 362-task, 17-template snapshot and distinguished it from the separate generation-time LLM pipeline. (§2.1; Appendix G.1. Reviewer 9w82.)

  3. Audited all 362 reference solutions and 2,046 checks, confirming that all seven reference failures belong to Release Gate and documenting unresolved target requirements. (§4.2.5; Appendices G.1–G.2. Reviewers VxgB and bNKA.)

  4. Added validator stress tests across 17 scenarios. Targeted city-binding checks reduce error-free negative accepts from 82/136 to 51/136 while retaining all 102 positive controls. (Appendix G.3, Tables 12–13. Reviewer bNKA.)

  5. Added 362/355/332-task sensitivity analyses and revised the Release Gate and shared-failure interpretation. (§§4.2.4–4.2.5, Table 5; Appendix G.2. Reviewers VxgB and bNKA.)

  6. Added scenario-cluster bootstrap, scenario-equal aggregation, merged-repair clustering, and leave-one-scenario-out analyses; adjacent rankings are no longer treated as resolved. (§4.2.1; Appendix G.4, Tables 14–15. Reviewers VxgB and bNKA.)

  7. Decomposed repair results by initial-state subtype and failed checks, including GPT-5.5's task-type contrast of 16/17 versus 0/25. (§4.3; Appendix G.5, Table 16. Reviewers VxgB and 9w82.)

  8. Added six-label definitions, a tag co-occurrence matrix, and a 17-scenario coverage table, clarifying overlapping demands and preservation controls. (§2.1; Appendix G.5, Tables 17–19. Reviewer 9w82.)

  9. Reproduced all 1,448 historical GPT outcomes through saved-action replay and updated the abstract, conclusion, and AI Use Statement to reflect the revised evidence and scope. (Appendix G.1; Abstract; Conclusion; AI Use Statement. All reviewers.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.