- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
AgentSWE tests whether coding agents can build agents that meet a natural-language requirement on cases they never saw. The benchmark is meant to measure the gap between what the the agent claim to build and what it actually built. Its 25 tasks span Creation (build from scratch), Editing (add features to production agent repos) and Optimization (improve a starter on BrowseComp, τ³ and three other benchmarks). Builders develop against 2 public cases, then the frozen delivery runs on held-out cases through a fixed runtime model. Creation and Editing are scored by validity gates plus an LLM-judged rubric, and Optimization by gap closure. All five builders score low (best 19.1 on Creation, 31.6 on Editing, under 5% gap closure on Optimization). The trajectory audit finds three failure modes: overfitting to visible cases, code that looks complete but crashes, and false completion claims.
Critical strengths
- A clean protocol for a question no existing benchmark measures. Freezing the delivery, hiding the held-out cases and fixing the runtime model isolate what the builder actually contributes.
Critical weaknesses
-
Solvability is never shown. The held-out cases were written by an LLM and not said to be tested for solvability. There is no reference implementation or human baseline, and several tasks never exceed about 20 in any setting. So low scores and the dev–held-out gap may partly reflect case difficulty or could be an edge cases. For example, task like PDF translation, if task specs never says what language pair needs to be supported, then testing on many low-resources pair is not a fair test.
-
The evidence is thin. Each task is scored on 6 cases. The author mentions the test are diverse but the paper presentation make it seem limited. The LLM judges share model families with the builders, and no human or cross-judge validation is reported.
-
misalignment with original motivation: after reading the paper, it remains unclear how the Editing section and Optimization section contributes to the original motivation of "Between judging built software and judging finished agents lies a step that current suites leave unmeasured: building the agent that a requirement asks for." (line 84-85). How is such gap measured is not clear to the reviewer. It seems the performance on some SWE benchmark suffice to test such gap. It's safe to assume the agent will handle the request and thus the performance on the hidden test suite will reflect the gap.
Questions for authors
- How many held-out cases per task did any run substantially solve? Can you run a reference or human-built agent on a subset of tasks?
- Is there any validation of the judge (human agreement, or cross-judge re-scoring)? Why is different model used for different tasks?
- How much do scores vary when a case is re-executed or re-judged?
- what does “clause” refers to? 6 test case per task seems limited rather than diverse
6 comments
W2 (judges), Q2: judge validation and choice of judges
Thank you for these questions. Below we report a cross-family validation of the judges and explain why the families use different judges. The number of cases per task is addressed under Q4.
Cross-family validation. We re-scored judged executions of the main comparison with Gemini 3.8 Flash, a judge from a model family that none of the builders or original judges belongs to, and had the original judges score the same outputs a second time (Table A). The cross-family judge ranks executions consistently with the original judges (Spearman 0.89 in Creation and 0.98 in Editing). In Creation, where the original judge, GPT-5.6 Sol, shares a family with two builders, the cross-family judge changes no builder's rank (the largest change in a builder's mean is 1.4 points), and the original judge shows no preference for builders of its own family (a difference of 0.5 points, with a 95% interval from −2.2 to 3.1). In Editing, where the original judge shares a family with the two DeepSeek builders, the cross-family judge agrees with the original judge about as closely as the original judge agrees with itself (a mean absolute difference of 2.8 against 2.6 points). No judge influences executions that fail the deterministic validity gates, which assign zero without a judge when the agent does not launch, exceeds its time budget or produces no parseable output, as in 58% of held-out Creation executions.
Table A. Agreement between judges, on the executions scored validly in both rounds (Creation: 95 for the cross-family comparison and 101 for the original judge's second scoring; Editing: 41).
| Creation | Editing | |
|---|---|---|
| Cross-family judge vs. original judge: Spearman | 0.89 | 0.98 |
| Cross-family judge vs. original judge: mean absolute difference (points) | 4.6 | 2.8 |
| Original judge scoring twice: Spearman | 0.93 | 0.98 |
| Original judge scoring twice: mean absolute difference (points) | 2.4 | 2.6 |
Why the families use different judges. Optimization is scored by the native evaluator of each source benchmark, and Creation and Editing each use one judge chosen for the evidence it must read. Creation is graded partly from rendered pages and screenshots, so the main comparison uses GPT-5.6 Sol. Editing evidence is entirely text, so Editing uses the lower-cost DeepSeek-V4.1-Flash. Using different judges across families does not affect the paper's conclusions, because scores are compared only within a family: the families differ in their tasks, runtime models and scoring, so their absolute levels are not comparable in any case.
Q3: variation under re-judging and re-execution
Thank you for this question. Below we report how much scores change when the same output is re-judged and when the same frozen delivery is re-executed, and how Optimization handles the variation of its native objectives.
Re-judging. When the original judge scores the same outputs a second time, scores change by 2.4 points on average in Creation and 2.6 in Editing, and the two scorings rank executions almost identically (Spearman 0.93 and 0.98; Table A under Q2).
Re-execution. We re-ran frozen deliveries on the same held-out cases in the same environment: 72 Creation pairs from 12 deliveries and 48 Editing pairs from 8 deliveries across 7 tasks (Table B). The validity gate gives the same outcome in 91.7% and 97.9% of re-executions, the two runs rank executions consistently (Spearman 0.93 and 0.97), and scores change by 7.4 and 4.8 points on average.
Table B. Re-executing frozen deliveries on the same cases.
| Creation | Editing | |
|---|---|---|
| Re-executed pairs | 72 (12 deliveries) | 48 (8 deliveries, 7 tasks) |
| Same validity-gate outcome | 91.7% | 97.9% |
| Spearman between the two runs | 0.93 | 0.97 |
| Mean absolute change (points) | 7.4 | 4.8 |
Optimization. Each Optimization run of the main comparison evaluates the starter and the selected candidate on the same held-out instances in the same run, so S compares them under the same conditions, and Table 10 reports per-instance wins and losses for every run. When the same starter is evaluated again, the per-instance outcome is the same on 76–90% of instances for the four benchmarks that score each instance as pass or fail; the paper therefore draws its Optimization conclusions over all 25 runs rather than from single runs.
Case-level variation therefore leaves gate outcomes and the ordering of executions largely unchanged, and the paper's conclusions are drawn over many cases and builders.
W2 (number of cases), Q4: clauses and diversity of cases
Thank you for these comments. We agree that the paper did not show clearly how much the held-out cases cover, and we have revised it accordingly. Below we explain what a clause is, how much of the requirement the held-out cases cover, and how the cases of a task differ.
What a clause is. A clause is one requirement stated in the task documents: something the agent must do, something it must deliver, or a limit it must respect. We now define it in Section 2.1.
Each case exercises many clauses of the requirement. In Creation and Editing, a held-out case is a complete instance of the job rather than a single test: it exercises 13 clauses on average in Creation and 23 in Editing, which the rubric scores as five to six separately weighted items. Together, the held-out cases exercise 96% of all requirement clauses (Table C). Optimization tasks hold out 25 to 100 instances of their source benchmark, and the suite has 365 held-out cases in total.
Table C. Requirement coverage of the held-out cases.
| Creation | Editing | |
|---|---|---|
| Requirement clauses | 145 | 310 |
| Clauses exercised per held-out case (mean) | 12.9 | 23.2 |
| Scored items per held-out case (mean) | 5.7 | 5.3 |
| Requirement clauses exercised by the held-out cases | 96% | 96% |
The cases of a task differ in inputs and situations. The six PDF translation cases, for example, differ in language pair (Spanish to English, and English to Japanese, German, Korean, Arabic and Traditional Chinese) and in layout, from a two-column paper with a paragraph split across pages to a figure with dense labels. The six OpenHands cases exercise different recovery situations, from durable recovery across three workspace crash boundaries to pause, resume and cancel followed by stale callbacks. Appendix L maps every case to the clauses it exercises, and every scored item traces to a clause stated in the requirement.
W3: how Editing and Optimization fit the motivation, and how the gap is measured
Thank you for this comment. The sentence you quote names only building an agent, while the paper also studies extending and improving it; we have revised that sentence to name all three and added the explanation below to Section 2.3.
What AgentSWE measures. The unmeasured step is delivering an agent that does the job its requirement describes. The builder receives the requirement and a few development cases; its delivery is then frozen and run end to end, with a fixed runtime model, on held-out cases that are new instances of the job, and graded against the requirement. The held-out score measures how faithfully the delivery serves the requirement, and its difference from the development score measures the gap between what the builder demonstrated and what it delivered (Section 2.2).
Why a SWE benchmark does not suffice. A SWE benchmark tests deterministic code against expected outputs fixed in advance. An agent's code is instead a harness around a model: its behavior appears only when the model runs inside it, its outputs often have no reference answer, and a harness can pass code-level checks, even unit tests that stub out the model, yet fail once a real model runs. For an agent, a hidden test suite must therefore run the frozen agent end to end on new instances of its job and judge what it does, which is what AgentSWE does. Whether the builder handled the request is thus measured rather than assumed, and in all three families the delivered agents fall well short of what the builders demonstrate during development.
Why Editing and Optimization. A commissioned agent keeps changing after delivery: users ask a coding agent to add capabilities or to improve the agent on its workload. Each change is again a commission, so the three families are one problem with different starting points (Table D), measured the same way and showing recurring failure modes. Editing in particular is not SWE-bench applied to an agent repository: the modified product runs end to end with a real model through scenarios such as crash recovery, and its behavior is judged, instead of its patch being checked only by unit tests.
Table D. One problem, three starting points.
| Family | Starts from | The requirement asks to | The frozen delivery runs on |
|---|---|---|---|
| Creation | An empty workspace | Build an agent | 6 unseen cases per task |
| Editing | A production agent | Add capabilities and keep existing behavior | 6 unseen cases per task |
| Optimization | A working agent | Raise a measurable objective | 25–100 unseen benchmark instances |
We thank the reviewer for the detailed review and for the concrete questions on solvability, the judges, score variation, case coverage and the motivation. We address each point below; all changes in the revised paper are marked in blue, and the new analyses are in Appendix L.
W1, Q1: solvability
Below we explain how we tested solvability with reference agents and how many cases were solved, what this implies for the low scores and the development–held-out gap, and the language pairs in PDF translation.
Every held-out Creation case has been solved. The held-out cases were drafted with a coding agent and then verified and refined by the authors (Section 3.1), and we have now also tested their solvability. As reference agents, we used two of the strongest current models in a general-purpose harness, Codex with GPT-6.1-Sol and Codex with GPT-6 Astra; on four cases of authorized vulnerability validation, Codex with DeepSeek-V4.1-Flash served as the reference agent. Each reference agent receives the same requirement as the builders and works on a held-out case directly. We count a case as solved when some agent's output on it scores at least 80. All six held-out cases of each of the ten Creation tasks have been solved: 23 of the 60 by agents delivered in our experiments, and the remaining 37 by the reference agents. This includes every case of the three tasks that stay below about 20 in all settings. Most reference solutions used a longer time budget or the evaluator's feedback (Appendix L lists the conditions for each case); they show that every case can be fully met from the stated requirement. Time is not what stops the delivered agents: only 15 of 270 held-out Creation executions reach the 600 s limit. In Editing, every one of the ten tasks also has at least one held-out case solved.
Low scores reflect the delivered agents, not impossible or edge cases. Every held-out Creation case can be solved, and every scored item traces to a clause that the requirement states. The Creation cases on which the builders' agents fail are therefore attainable cases within the requirement, and the low scores and the gap between development and held-out results reflect the agents that were delivered.
PDF translation. The requirement asks the agent to handle whatever source and target languages a request names: it requires fonts and text direction suited to the target script and forbids hard-coding languages. The held-out pairs are Spanish to English and English to Japanese, German, Korean, Arabic and Traditional Chinese, none of them low-resource, and agents delivered in our experiments solved all six cases within the main budget of 600 s per case.