acceptodds
Under review as a conference paper at ICLR 2027

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

22 views

Abstract

As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, turning verdicts into signals by which coding agents may learn to build the agents they are asked for.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Agentic Software Engineering, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_u7nNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

AgentSWE tests whether coding agents can build agents that meet a natural-language requirement on cases they never saw. The benchmark is meant to measure the gap between what the the agent claim to build and what it actually built. Its 25 tasks span Creation (build from scratch), Editing (add features to production agent repos) and Optimization (improve a starter on BrowseComp, τ³ and three other benchmarks). Builders develop against 2 public cases, then the frozen delivery runs on held-out cases through a fixed runtime model. Creation and Editing are scored by validity gates plus an LLM-judged rubric, and Optimization by gap closure. All five builders score low (best 19.1 on Creation, 31.6 on Editing, under 5% gap closure on Optimization). The trajectory audit finds three failure modes: overfitting to visible cases, code that looks complete but crashes, and false completion claims.

Critical strengths
  1. A clean protocol for a question no existing benchmark measures. Freezing the delivery, hiding the held-out cases and fixing the runtime model isolate what the builder actually contributes.
Critical weaknesses
  1. Solvability is never shown. The held-out cases were written by an LLM and not said to be tested for solvability. There is no reference implementation or human baseline, and several tasks never exceed about 20 in any setting. So low scores and the dev–held-out gap may partly reflect case difficulty or could be an edge cases. For example, task like PDF translation, if task specs never says what language pair needs to be supported, then testing on many low-resources pair is not a fair test.

  2. The evidence is thin. Each task is scored on 6 cases. The author mentions the test are diverse but the paper presentation make it seem limited. The LLM judges share model families with the builders, and no human or cross-judge validation is reported.

  3. misalignment with original motivation: after reading the paper, it remains unclear how the Editing section and Optimization section contributes to the original motivation of "Between judging built software and judging finished agents lies a step that current suites leave unmeasured: building the agent that a requirement asks for." (line 84-85). How is such gap measured is not clear to the reviewer. It seems the performance on some SWE benchmark suffice to test such gap. It's safe to assume the agent will handle the request and thus the performance on the hidden test suite will reflect the gap.

Questions for authors
  1. How many held-out cases per task did any run substantially solve? Can you run a reference or human-built agent on a subset of tasks?
  2. Is there any validation of the judge (human agreement, or cross-judge re-scoring)? Why is different model used for different tasks?
  3. How much do scores vary when a case is re-executed or re-judged?
  4. what does “clause” refers to? 6 test case per task seems limited rather than diverse
6 comments

We thank the reviewer for the detailed review and for the concrete questions on solvability, the judges, score variation, case coverage and the motivation. We address each point below; all changes in the revised paper are marked in blue, and the new analyses are in Appendix L.

W1, Q1: solvability

Below we explain how we tested solvability with reference agents and how many cases were solved, what this implies for the low scores and the development–held-out gap, and the language pairs in PDF translation.

Every held-out Creation case has been solved. The held-out cases were drafted with a coding agent and then verified and refined by the authors (Section 3.1), and we have now also tested their solvability. As reference agents, we used two of the strongest current models in a general-purpose harness, Codex with GPT-6.1-Sol and Codex with GPT-6 Astra; on four cases of authorized vulnerability validation, Codex with DeepSeek-V4.1-Flash served as the reference agent. Each reference agent receives the same requirement as the builders and works on a held-out case directly. We count a case as solved when some agent's output on it scores at least 80. All six held-out cases of each of the ten Creation tasks have been solved: 23 of the 60 by agents delivered in our experiments, and the remaining 37 by the reference agents. This includes every case of the three tasks that stay below about 20 in all settings. Most reference solutions used a longer time budget or the evaluator's feedback (Appendix L lists the conditions for each case); they show that every case can be fully met from the stated requirement. Time is not what stops the delivered agents: only 15 of 270 held-out Creation executions reach the 600 s limit. In Editing, every one of the ten tasks also has at least one held-out case solved.

Low scores reflect the delivered agents, not impossible or edge cases. Every held-out Creation case can be solved, and every scored item traces to a clause that the requirement states. The Creation cases on which the builders' agents fail are therefore attainable cases within the requirement, and the low scores and the gap between development and held-out results reflect the agents that were delivered.

PDF translation. The requirement asks the agent to handle whatever source and target languages a request names: it requires fonts and text direction suited to the target script and forbids hard-coding languages. The held-out pairs are Spanish to English and English to Japanese, German, Korean, Arabic and Traditional Chinese, none of them low-resource, and agents delivered in our experiments solved all six cases within the main budget of 600 s per case.

W2 (judges), Q2: judge validation and choice of judges

Thank you for these questions. Below we report a cross-family validation of the judges and explain why the families use different judges. The number of cases per task is addressed under Q4.

Cross-family validation. We re-scored judged executions of the main comparison with Gemini 3.8 Flash, a judge from a model family that none of the builders or original judges belongs to, and had the original judges score the same outputs a second time (Table A). The cross-family judge ranks executions consistently with the original judges (Spearman 0.89 in Creation and 0.98 in Editing). In Creation, where the original judge, GPT-5.6 Sol, shares a family with two builders, the cross-family judge changes no builder's rank (the largest change in a builder's mean is 1.4 points), and the original judge shows no preference for builders of its own family (a difference of 0.5 points, with a 95% interval from −2.2 to 3.1). In Editing, where the original judge shares a family with the two DeepSeek builders, the cross-family judge agrees with the original judge about as closely as the original judge agrees with itself (a mean absolute difference of 2.8 against 2.6 points). No judge influences executions that fail the deterministic validity gates, which assign zero without a judge when the agent does not launch, exceeds its time budget or produces no parseable output, as in 58% of held-out Creation executions.

Table A. Agreement between judges, on the executions scored validly in both rounds (Creation: 95 for the cross-family comparison and 101 for the original judge's second scoring; Editing: 41).

CreationEditing
Cross-family judge vs. original judge: Spearman0.890.98
Cross-family judge vs. original judge: mean absolute difference (points)4.62.8
Original judge scoring twice: Spearman0.930.98
Original judge scoring twice: mean absolute difference (points)2.42.6

Why the families use different judges. Optimization is scored by the native evaluator of each source benchmark, and Creation and Editing each use one judge chosen for the evidence it must read. Creation is graded partly from rendered pages and screenshots, so the main comparison uses GPT-5.6 Sol. Editing evidence is entirely text, so Editing uses the lower-cost DeepSeek-V4.1-Flash. Using different judges across families does not affect the paper's conclusions, because scores are compared only within a family: the families differ in their tasks, runtime models and scoring, so their absolute levels are not comparable in any case.


Q3: variation under re-judging and re-execution

Thank you for this question. Below we report how much scores change when the same output is re-judged and when the same frozen delivery is re-executed, and how Optimization handles the variation of its native objectives.

Re-judging. When the original judge scores the same outputs a second time, scores change by 2.4 points on average in Creation and 2.6 in Editing, and the two scorings rank executions almost identically (Spearman 0.93 and 0.98; Table A under Q2).

Re-execution. We re-ran frozen deliveries on the same held-out cases in the same environment: 72 Creation pairs from 12 deliveries and 48 Editing pairs from 8 deliveries across 7 tasks (Table B). The validity gate gives the same outcome in 91.7% and 97.9% of re-executions, the two runs rank executions consistently (Spearman 0.93 and 0.97), and scores change by 7.4 and 4.8 points on average.

Table B. Re-executing frozen deliveries on the same cases.

CreationEditing
Re-executed pairs72 (12 deliveries)48 (8 deliveries, 7 tasks)
Same validity-gate outcome91.7%97.9%
Spearman between the two runs0.930.97
Mean absolute change (points)7.44.8

Optimization. Each Optimization run of the main comparison evaluates the starter and the selected candidate on the same held-out instances in the same run, so S compares them under the same conditions, and Table 10 reports per-instance wins and losses for every run. When the same starter is evaluated again, the per-instance outcome is the same on 76–90% of instances for the four benchmarks that score each instance as pass or fail; the paper therefore draws its Optimization conclusions over all 25 runs rather than from single runs.

Case-level variation therefore leaves gate outcomes and the ordering of executions largely unchanged, and the paper's conclusions are drawn over many cases and builders.

W2 (number of cases), Q4: clauses and diversity of cases

Thank you for these comments. We agree that the paper did not show clearly how much the held-out cases cover, and we have revised it accordingly. Below we explain what a clause is, how much of the requirement the held-out cases cover, and how the cases of a task differ.

What a clause is. A clause is one requirement stated in the task documents: something the agent must do, something it must deliver, or a limit it must respect. We now define it in Section 2.1.

Each case exercises many clauses of the requirement. In Creation and Editing, a held-out case is a complete instance of the job rather than a single test: it exercises 13 clauses on average in Creation and 23 in Editing, which the rubric scores as five to six separately weighted items. Together, the held-out cases exercise 96% of all requirement clauses (Table C). Optimization tasks hold out 25 to 100 instances of their source benchmark, and the suite has 365 held-out cases in total.

Table C. Requirement coverage of the held-out cases.

CreationEditing
Requirement clauses145310
Clauses exercised per held-out case (mean)12.923.2
Scored items per held-out case (mean)5.75.3
Requirement clauses exercised by the held-out cases96%96%

The cases of a task differ in inputs and situations. The six PDF translation cases, for example, differ in language pair (Spanish to English, and English to Japanese, German, Korean, Arabic and Traditional Chinese) and in layout, from a two-column paper with a paragraph split across pages to a figure with dense labels. The six OpenHands cases exercise different recovery situations, from durable recovery across three workspace crash boundaries to pause, resume and cancel followed by stale callbacks. Appendix L maps every case to the clauses it exercises, and every scored item traces to a clause stated in the requirement.


W3: how Editing and Optimization fit the motivation, and how the gap is measured

Thank you for this comment. The sentence you quote names only building an agent, while the paper also studies extending and improving it; we have revised that sentence to name all three and added the explanation below to Section 2.3.

What AgentSWE measures. The unmeasured step is delivering an agent that does the job its requirement describes. The builder receives the requirement and a few development cases; its delivery is then frozen and run end to end, with a fixed runtime model, on held-out cases that are new instances of the job, and graded against the requirement. The held-out score measures how faithfully the delivery serves the requirement, and its difference from the development score measures the gap between what the builder demonstrated and what it delivered (Section 2.2).

Why a SWE benchmark does not suffice. A SWE benchmark tests deterministic code against expected outputs fixed in advance. An agent's code is instead a harness around a model: its behavior appears only when the model runs inside it, its outputs often have no reference answer, and a harness can pass code-level checks, even unit tests that stub out the model, yet fail once a real model runs. For an agent, a hidden test suite must therefore run the frozen agent end to end on new instances of its job and judge what it does, which is what AgentSWE does. Whether the builder handled the request is thus measured rather than assumed, and in all three families the delivered agents fall well short of what the builders demonstrate during development.

Why Editing and Optimization. A commissioned agent keeps changing after delivery: users ask a coding agent to add capabilities or to improve the agent on its workload. Each change is again a commission, so the three families are one problem with different starting points (Table D), measured the same way and showing recurring failure modes. Editing in particular is not SWE-bench applied to an agent repository: the modified product runs end to end with a real model through scenarios such as crash recovery, and its behavior is judged, instead of its patch being checked only by unit tests.

Table D. One problem, three starting points.

FamilyStarts fromThe requirement asks toThe frozen delivery runs on
CreationAn empty workspaceBuild an agent6 unseen cases per task
EditingA production agentAdd capabilities and keep existing behavior6 unseen cases per task
OptimizationA working agentRaise a measurable objective25–100 unseen benchmark instances
Reviewer_64idOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies no existing benchmark that tests a quiet agent assumption, an agent fits to the user's full requirements.

This paper proposes AgentSWE, a benchmark for agent software engineering for challenging the quiet assumption by evaluating tasks as commission instead of exam.

Experimental results demonstrate that agents often fail to deliver what a requirement asks for.

Analysis was conducted to identify the failures modes for a future coding agent direction

Critical strengths

The paper reveals that coding agents are fully serving the user’s requirement by introducing AgentSWE for real-world commissions to test the full development cycle.

AgentSWE finds that current agents got low scores for 3 stages.

Empirical analysis uncover the recurring failure modes, e.g., overfitting to visible cases, hollow runtimes that seem complete but do not work, and unsupported claims of completion for improvements.

Critical weaknesses

AgentSWE tasks are too strict since scores are too low for current agents, which may not be useful for distinguishing current agents. Even the popular GPT-5.6 has very low scores and has a 0 score on some tasks as Qwen3.6 in Table 1. But in practice, GPT-5.6 is much more useful than Qwen3.6 for coding agents. The benchmark may be misleading.

Not clear how useful the normalized score is if it is less than 0 (Eq. 3).

Lack the cost of experimental results for evaluating an agent on AgentSWE compared to traditional benchmarks such as SWE-bench and how helpful it can provide in real practice.

Task sample size is quite small, only 25 tasks. Not sure how this limits the evaluation of current agents.

Questions for authors

See Weaknesses

3 comments

We thank the reviewer for the careful review and for the constructive comments on the difficulty of the tasks, the normalized score, cost and practical value, and the number of tasks. We address each point below; all changes in the revised paper are marked in blue, and the new analyses are in Appendix L.

W1: too strict, or misleading?

Below we explain why AgentSWE is designed to be challenging, how it still separates GPT-5.6 Sol from Qwen3.6, and why its tasks are attainable.

AgentSWE is designed to challenge frontier coding agents. Its purpose is to test whether a coding agent can deliver an agent that serves a requirement on cases it has not seen, and to stay informative as coding agents improve. Low scores show that this capability is far from solved, which leaves the benchmark room to track progress; SWE-bench was similarly hard when it was introduced, with the best model at the time resolving 1.96% of issues. The difficulty lies in delivery rather than in strict grading: most Creation zeros arise because the delivered agent crashes, exceeds its time budget or produces no parseable output on an input it has not seen, as in 58% of held-out Creation executions, while an agent that runs receives partial credit item by item.

AgentSWE separates builders. A zero on an individual task is a task-level outcome; across Creation and Editing the difference is clear. In Table 1, GPT-5.6 Sol averages 13.7 on Creation and 23.9 on Editing, against 2.3 and 0.0 for Qwen3.6, which submits no Editing candidate at all. Among held-out Creation executions that pass the validity gates, GPT-5.6 Sol scores about four times as high as Qwen3.6. We also added a supplementary experiment, AgentSWE-Lite, on eight of the 25 tasks: each of three builders builds every task three times independently, and GPT-5.6 Sol scores above Qwen3.6 on all eight tasks (Table A). The benchmark therefore agrees with the practical impression that GPT-5.6 Sol is a much stronger coding agent than Qwen3.6.

Table A. GPT-5.6 Sol and Qwen3.6 in the main comparison and in AgentSWE-Lite.

GPT-5.6 SolQwen3.6
Creation, mean held-out Result (Table 1)13.72.3
Editing, mean held-out Result (Table 1)23.90.0
Creation, mean Result of held-out executions that pass the validity gates47.611.7
AgentSWE-Lite tasks with the higher mean over three builds8 of 80 of 8

The tasks are attainable. With the agents delivered in our experiments and, as reference agents, Codex with GPT-6.1-Sol and Codex with GPT-6 Astra (and, on four cases of one task, Codex with DeepSeek-V4.1-Flash), we have solved (scored at least 80 on) every one of the 60 held-out Creation cases, some with a longer time budget or with the evaluator's feedback, and at least one held-out case of each of the ten Editing tasks (Appendix L). The low scores therefore show what current builders deliver, not that the tasks are impossible: even a builder that reaches high development scores often delivers an agent that fails on unseen inputs, and this is the gap AgentSWE is designed to expose.


W2: negative normalized score

Thank you for the question. The normalized score of Eq. 3, which we report as S, is the share of the remaining gap to the target that the delivered agent closes on held-out instances, so a negative S means that the optimized agent scores below the starter it began from. We keep these values rather than clipping them at zero because a regression is a real outcome of optimization that a user needs to see.

W3: cost and practical value

Thank you for this suggestion. We give the cost first and then the practical value.

Cost. For evaluating a builder, we recommend AgentSWE-Lite, eight of the 25 tasks covering all three families. With GPT-5.6 Sol as the builder, one build of all eight Lite tasks costs about USD 215 in total, including the runtime model and judge, while one pass over SWE-bench Verified with the same model costs about USD 575. With one build per task, evaluating a builder on AgentSWE-Lite therefore costs well under one pass over SWE-bench Verified, and with three builds about as much (Table B).

Table B. Cost of evaluating GPT-5.6 Sol as a builder. AgentSWE-Lite costs include DeepSeek-V4.1-Flash as the runtime model and judge; the SWE-bench Verified cost is from the Vals.ai leaderboard (mini-SWE-agent harness, USD 1.15 per instance).

EvaluationCost (USD)
AgentSWE-Lite (8 tasks), one build per task≈ 215
AgentSWE-Lite (8 tasks), three builds per task≈ 640
SWE-bench Verified, 500 instances≈ 575

Practical value. AgentSWE helps in three ways.

  • Choosing a builder. Whenever a coding agent is asked to create, extend or optimize an agent, AgentSWE shows which builder delivers agents that work on inputs the builder never saw (Table A under W1), something neither a builder's development score nor its completion report reliably indicates.
  • Knowing the limits of coding agents. The failure modes we identify chart the capability boundary of current coding agents: they overfit to the visible cases, ship code that looks complete but does no work, and claim completions that their own records contradict.
  • Improving coding agents at scale. Each family follows a documented construction procedure, so tasks can be built from further open-source systems and benchmarks in the same way, providing the environments needed to train and evaluate coding agents on delivering agents.

W4: what 25 tasks support

Thank you for raising this. In short, the number of tasks does not limit the evaluation of current agents: each task contributes many reproducible measurements, and the 25 tasks suffice both to separate builders and to support the paper's findings. Below we explain these three points and how the task set will grow.

Each task contributes many reproducible measurements. An AgentSWE task is a full commission rather than a single issue: the builder develops for hours, and the delivered agent is then run on every held-out case. The 25 tasks contain 365 held-out cases, and each Creation or Editing case exercises many requirement clauses, 13 on average in Creation and 23 in Editing, scored as five to six separately weighted items. Benchmarks of long-horizon work are of similar size for the same reason: PaperBench consists of 20 papers and RE-Bench of seven environments. The measurements are also reproducible: re-executing frozen deliveries gives the same validity-gate outcome in over 90% of executions and yields strongly correlated scores, and re-scoring the same output changes the score by 2 to 3 points on average (Table C).

Table C. Reproducibility of the measurement.

CreationEditing
Re-execution: same validity-gate outcome91.7%97.9%
Re-execution: Spearman correlation between the two runs0.930.97
Re-execution: mean absolute change (points)7.44.8
Re-scoring the same output: mean absolute change (points)2.42.6

The tasks separate strong builders from weak ones. On AgentSWE-Lite, where every pairing is built three times, GPT-5.6 Sol and DeepSeek-V4.1-Flash each score above Qwen3.6 on all eight tasks, consistent with the main comparison (Table A under W1).

The findings hold across conditions. The paper's empirical conclusions are that held-out performance is low and that it falls below development performance. AgentSWE-Lite reproduces both findings with three builds per task and, in Creation and Optimization, a different runtime model and judge, as does an extended-budget setting that changes the builder, runtime model, judge, case budget and requirement on all ten Creation tasks (Table D); varying the number of development cases, adding prompt priors or changing the harness also leaves held-out Result low. The failure modes recur as well: in the extended-budget setting a third of held-out executions still produce no valid result, and on AgentSWE-Lite two GPT-5.6 Sol builds of database analytics reach 99 on the development cases but only 11.0 and 6.5 on held-out cases.

Table D. The main findings under different conditions of the build.

SettingChanged from the main comparisonMean held-out ResultDevelopment vs. held-out
Main comparison, Creation–11.2Runs ending development at ≥ 90 average 9.9 held-out
Main comparison, Editing–0.0–31.6 across buildersOne delivery scores 100 on development and 28.3 held-out (Section 4.3)
Extended budget, 10 Creation tasksBuilder, runtime model, judge, case budget, requirement31.5Development higher on 10 of 10 tasks
AgentSWE-Lite, 6 Creation and Editing tasksBuilders; Creation runtime model and judge; three builds per task0.0–33.5 across buildersDevelopment higher in 16 of the 19 Creation builds with a development score

Growing the task set. Each family follows a documented construction procedure, in which candidate tasks are screened and every draft is verified by the authors (Section 3.1), so new tasks can be added in the same way. The number of tasks is bounded mainly by evaluation cost, and with one build per task, AgentSWE-Lite keeps that cost below one pass over SWE-bench Verified (see W3). We will keep extending the task set.

Reviewer_wg77OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduce AgentSWE, a benchmark evaluates how well coding agent does during full lifecycle of software development. The empirical results reveal the gap between what a builder demonstrates and what a coding agent delivers, and the work indentifies some recurring failure modes.

Critical strengths

The evaluation goal is targetted at real-world software engineering demand, and the work designs a solid evaluation pipeline, the work has practical significance and specific diagnosis values.

Critical weaknesses
  • On first read of the abstract, the narrative of full lifecycle coverage seems to be compelling. But it later became clear that individual tasks do not span creation, editing, and optimization. Instead, the benchmark collects separate tasks for each stage. Given that relevant benchmarks already exist for each of these stages, this substantially weakens the paper’s contribution. The abstract should make this distinction clearer to avoid unnecessary misunderstandings about the scope of the evaluation.

  • Creation, Editing, and Optimization contain only 1010, 1010, and 55 tasks respectively. Such a limited evaluation, which also lacks scalability, introduces substantial variance into the assessment of agents. Consequently, both the reported scores and the resulting failure analyses rest on a highly biased empirical foundation.

Questions for authors

please address the weaknesses.

5 comments

We thank the reviewer for the careful reading and for the constructive comments on the scope of AgentSWE and the reliability of its evaluation. We address each point below; all changes in the revised paper are marked in blue, and the new analyses are in Appendix L.

W1: scope of "full lifecycle"

By "lifecycle" we mean that AgentSWE covers all three stages, with a separate task family for each; each task evaluates one stage. We have revised the abstract and the introduction to state this explicitly. Below we explain why the stages are evaluated as separate families, how AgentSWE differs from existing benchmarks, and what evaluating the three stages together reveals.

Why separate task families. Each stage starts from a controlled artifact: an empty workspace for Creation, a pinned production agent for Editing, and a runnable starter agent for Optimization. If one task chained the stages, a failed creation would leave editing and optimization to work on an agent that does not run, and failures could no longer be attributed to a stage. A failed creation would be the common case: 58% of held-out Creation executions fail a validity gate, crashing, timing out or producing no parseable output (Section 4.3).

How AgentSWE differs from existing benchmarks. Existing benchmarks cover neighboring settings. Those that build software from requirements, such as NL2Repo-Bench and PaperBench, and those that repair it, such as SWE-bench, check ordinary code against tests or an answer known in advance. Agent benchmarks such as AgentBench evaluate an agent that someone else built. The closest work, AIE-Bench, has a meta-agent improve a pinned agent scaffold to raise a benchmark score; it resembles our Optimization family but neither starts from a natural-language requirement nor measures the gap between development and delivery (Table A). For Creation and Editing, no existing benchmark has a coding agent build or extend an agent and then runs what it delivers. Across all three stages, AgentSWE gives a general coding agent a natural-language requirement with no reference solution and only a few development cases. It then freezes what the builder delivers and runs it on held-out cases the builder has never seen, which exercise requirement clauses the development cases leave out. This measures the gap between what a builder demonstrates and what it delivers, which is what a user who delegates the work needs to know.

Table A. Properties of the closest benchmarks. Benchmarks for ordinary software start from a requirement but deliver code checked against a reference; AIE-Bench delivers an agent but optimizes a score. AgentSWE is the only one that combines all five properties.

BenchmarkStarts from a natural-language requirementDelivers an agentGraded without a reference solutionMeasures the gap between development and deliveryCovers creating, editing and optimizing agents
SWE-bench✓––––
NL2Repo-Bench✓––––
PaperBench✓––––
AIE-Bench–✓✓–Optimizing only
AgentSWE✓✓✓✓✓

What evaluating the three stages together reveals. Under one protocol, the same three failure modes recur across the stages: development results overstate delivered results in Creation, Editing and Optimization alike; code that looks complete does no work when run, from created agents that crash on new inputs to added capabilities that are never invoked; and both builders and the agents they deliver claim completions that their own records contradict. These are properties of the builder rather than of one stage, and they become visible only because all three stages are tested the same way. Tasks that chain the stages are a natural extension that we leave to future work.

W2: number of tasks, variance, and the failure analysis

Thank you for raising this. The concern runs from the number of tasks, through variance, to the reliability of the scores and of the failure analysis; we address each step below, and then scalability.

The scores rest on many reproducible measurements. The 25 tasks contain 365 held-out cases, and each Creation or Editing case exercises many requirement clauses, 13 on average in Creation and 23 in Editing, scored as five to six separately weighted items. The measurement itself is stable: re-executing frozen deliveries gives the same validity-gate outcome in over 90% of executions and yields strongly correlated scores, and re-scoring the same output changes the score by 2 to 3 points on average (Table B).

Table B. Reproducibility of the measurement.

CreationEditing
Re-execution: same validity-gate outcome91.7%97.9%
Re-execution: Spearman correlation between the two runs0.930.97
Re-execution: mean absolute change (points)7.44.8
Re-scoring the same output: mean absolute change (points)2.42.6

The findings hold across conditions. The paper's empirical conclusions are that held-out performance is low and that it falls below development performance. We added a supplementary experiment, AgentSWE-Lite, in which each of three builders builds eight of the 25 tasks three times independently, with a different runtime model and judge in Creation and Optimization. It reproduces both findings, as does an extended-budget setting that changes the builder, runtime model, judge, case budget and requirement on all ten Creation tasks (Table C); varying the number of development cases, adding prompt priors or changing the harness also leaves held-out Result low. AgentSWE-Lite also separates builders: GPT-5.6 Sol and DeepSeek-V4.1-Flash each score above Qwen3.6 on all eight tasks.

Table C. The main findings under different conditions of the build.

SettingChanged from the main comparisonMean held-out ResultDevelopment vs. held-out
Main comparison, Creation–11.2Runs ending development at ≥ 90 average 9.9 held-out
Main comparison, Editing–0.0–31.6 across buildersOne delivery scores 100 on development and 28.3 held-out (Section 4.3)
Extended budget, 10 Creation tasksBuilder, runtime model, judge, case budget, requirement31.5Development higher on 10 of 10 tasks
AgentSWE-Lite, 6 Creation and Editing tasksBuilders; Creation runtime model and judge; three builds per task0.0–33.5 across buildersDevelopment higher in 16 of the 19 Creation builds with a development score

The failure analysis does not rest on score averages. Each failure mode is established from records, namely the builders' trajectories, the frozen artifacts and the execution traces, and each recurs across stages and builders. The modes also recur under the supplementary conditions: in the extended-budget setting a third of held-out executions still produce no valid result, and in AgentSWE-Lite the overfitting we trace on database analytics appears again. Score variance therefore does not bias the failure analysis, which reports observed behavior rather than estimates.

Scalability. Each family follows a documented construction procedure: a coding agent drafts each task from an open-source agent, a production codebase or an established benchmark, and the authors screen the candidates and verify every draft (Section 3.1); every scored item traces to a requirement clause. The number of tasks is bounded by evaluation cost rather than by construction, since each task builds an agent over several hours before running it. AgentSWE-Lite offers a low-cost entry point, and we will keep extending the task set.

The authors' response addressed most of my concerns and i have also read the other reviewers’ comments, which I consider this a valuable contribution, though I’m still unsure whether to raise my score.

Since the rebuttal period is drawing to a close, I’d like to just leave the authors with an open question: benchmarks such as Frontier SWE and Deep SWE show how SWE benchmarks have evolved. Do you think there is a natural endpoint to that evolution? i would happy to get some insightful opnitions from the authors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

general response

General Response

We thank all reviewers for their careful reading and constructive comments. Below we restate what AgentSWE measures, how its tasks are built and why it is useful, and then address three main concerns.

What AgentSWE measures. AgentSWE measures whether an agent delivered by a coding agent does the job its requirement describes on inputs the builder never saw. A delivered agent is later modified or optimized, and each modification or optimization is again a commission, so Creation, Editing and Optimization are three starting points of one problem: in each family, the builder works from a natural-language requirement and a few development cases, and its frozen delivery runs end to end on held-out cases.

How the tasks are built. Each family derives its tasks from real systems. A Creation task reverse-engineers a mature open-source domain agent into a self-contained requirement, extended with capabilities beyond the original; an Editing task commissions new capabilities in the pinned codebase of a production agent; and an Optimization task starts from the runnable starter agent of an established benchmark. Because every task is grounded in a working system, its requirement is realistic; in addition, the authors verify every draft to ensure the quality of each task. The tasks span diverse domains: Creation ranges from desktop GUI automation to formal theorem proving, Editing covers ten production agents such as OpenHands, Aider and Codex CLI, and Optimization spans five benchmarks from web research to computer use. Each held-out Creation or Editing case in turn exercises many requirement clauses, spanning functionality, interfaces and constraints.

Practical value. AgentSWE is useful in three ways. First, it helps choose a builder: whenever a coding agent is asked to create, extend or optimize an agent, AgentSWE shows which builder delivers agents that work on unseen inputs, something neither development scores nor completion reports reliably indicate. Second, the failure modes it reveals, such as overfitting to the visible cases and claiming completions that the builders' own records contradict, chart the capability boundary of current coding agents. Third, because the construction method scales, AgentSWE can grow into the environments needed to train and evaluate coding agents on delivering agents, and thereby to improve this capability.

Main concerns

Are 25 tasks enough, and are the results stable? (Reviewers wg77, 64id, u7nN) The number of tasks does not limit the conclusions, for three reasons. First, each task yields many measurements: the 25 tasks contain 365 held-out cases, and each Creation or Editing case exercises many requirement clauses. Second, each measurement is reproducible: re-executing frozen deliveries gives the same validity-gate outcome in over 90% of executions, and re-scoring the same output changes the score by 2 to 3 points on average. Third, the conclusions do not depend on one configuration: on AgentSWE-Lite, a supplementary experiment in which three builders each build eight of the tasks three times, and in an extended-budget setting, held-out performance stays low and below development performance, and the failure modes recur, such as overfitting to the visible cases.

Do the low scores carry information? (Reviewers 64id, u7nN) Yes: the scores are low by design, they reflect the delivered agents rather than strict grading, and they still separate builders. This difficulty keeps the benchmark informative as coding agents improve; SWE-bench was similarly hard when it was introduced. The difficulty lies in delivery: 58% of held-out Creation executions fail a validity gate on inputs the builder never saw, although every held-out Creation case has been solved by a delivered or reference agent. And the scores separate builders: on AgentSWE-Lite, GPT-5.6 Sol and DeepSeek-V4.1-Flash each score above Qwen3.6 on all eight tasks.

Are the judges reliable? (Reviewer u7nN) A judge from an unrelated model family (Gemini 3.8 Flash) ranks executions consistently with the original judges (Spearman 0.89 in Creation and 0.98 in Editing) and changes no Creation builder's rank, and executions that fail a deterministic validity gate never reach a judge.

Revisions. The revision, marked in blue, clarifies the scope and motivation (abstract, introduction and Section 2), adds the clause coverage, solvability and choice of judges to Section 3 and AgentSWE-Lite to Section 4.4, and reports the new analyses in Appendix L.

AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers for their careful reviews and their engagement during the discussion. Below we summarize the strengths they recognized and how we addressed the main concerns.

Strengths recognized by the reviewers

  • Reviewer wg77 found the evaluation goal targeted at real-world software engineering demand and the evaluation pipeline solid, and noted that the work has "practical significance and specific diagnosis values". After the rebuttal, the reviewer stated that our response "addressed most of my concerns" and considers AgentSWE "a valuable contribution".
  • Reviewer 64id highlighted that AgentSWE tests coding agents on real-world commissions across the full development cycle, and that its empirical analysis uncovers recurring failure modes: "overfitting to visible cases, hollow runtimes that seem complete but do not work, and unsupported claims of completion".
  • Reviewer u7nN described AgentSWE as "a clean protocol for a question no existing benchmark measures", noting that "freezing the delivery, hiding the held-out cases and fixing the runtime model isolate what the builder actually contributes". After we showed a reference solution for every Creation and Editing task, the reviewer raised the score.

Main concerns and how we addressed them

  • Scope and motivation (wg77, u7nN). We revised the abstract and the introduction to state that each task evaluates one stage, and revised the motivation to cover building, extending and improving an agent. We also explained why the stages form separate task families and how AgentSWE differs from existing benchmarks.
  • Number of tasks and cases, and stability (wg77, 64id, u7nN). The 25 tasks contain 365 held-out cases, and each Creation or Editing case exercises many requirement clauses. Re-executing frozen deliveries gives the same validity-gate outcome in over 90% of executions, and re-scoring the same output changes the score by 2 to 3 points on average. A supplementary experiment, AgentSWE-Lite, in which three builders each build eight of the tasks three times, and an extended-budget setting reproduce the main findings: held-out performance stays low and below development performance.
  • Low scores (64id, u7nN). The difficulty lies in delivery rather than in strict grading, and the scores still separate builders: on AgentSWE-Lite, GPT-5.6 Sol and DeepSeek-V4.1-Flash each score above Qwen3.6 on all eight tasks.
  • Solvability (u7nN). Every Creation and Editing task has a reference solution that scores at least 80 on all six of its held-out cases; in Editing, the reference is one frozen delivery run once on all six cases.
  • Judges (u7nN). A judge from an unrelated model family ranks executions consistently with the original judges (Spearman 0.89 in Creation and 0.98 in Editing).
  • Development vs. held-out in Editing (u7nN). Over the 33 Editing runs that deliver, the mean last development score is 49.0 and the mean held-out Result is 30.6.
  • Cost and normalized score (64id). With GPT-5.6 Sol as the builder, one build of the eight AgentSWE-Lite tasks costs about USD 215, less than one pass over SWE-bench Verified (about USD 575). A negative normalized score means that the optimized agent scores below its starter; we keep these values rather than clipping them at zero.

The revisions are marked in blue in the revised paper, with the new analyses in Appendix L. We thank the reviewers again for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.