acceptodds
Under review as a conference paper at ICLR 2027

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

Abstract

Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent, LLM Evaluation, Long Horizon

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BkzyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

E2E-SWE evaluates whether coding agents can construct complete, installable repositories from natural-language specifications in an empty workspace. The benchmark contains 186 tasks spanning 11 programming languages, grounded in public GitHub repositories. Software engineers collaborate with LLMs to author public-interface hidden tests and corresponding specifications, followed by static audits and rollout-guided task refinement. Agents work offline, and their generated repositories are installed and graded in fresh containers. The authors evaluate 13 models over four independent full-corpus runs per model, reporting pass@1 from 11.7% to 67.7%. Additional analyses examine reasoning effort, generated code size, the distribution of reasoning across turns, residual specification–test defects, and overlap with reference code.

Critical strengths
  1. An important evaluation target with a well-motivated approach to test fairness. Whole-repository generation makes specification–test mismatches particularly consequential because a single unsupported assertion can fail an otherwise correct implementation. Authoring tests around public behavior and aligning the specification with those tests directly addresses this problem, while allowing agents to choose their internal architecture. This gives the benchmark a useful focus on implementing a stated behavioral contract.

  2. A substantial evaluation effort with a sensible execution protocol. The benchmark covers 186 tasks across 11 languages and evaluates 13 models with four independent runs per model. Offline generation and installation in a fresh grading container reduce dependence on incidental development state. The wide observed performance range, reasoning-effort experiments, and explicit reporting of residual task issues and reference-code overlap provide useful evidence about current repository-generation systems.

Critical weaknesses
  1. The central claim of reliable specification–test alignment needs stronger independent validation. Section 2.3 does not specify the models and stopping criteria used in refinement, when tasks were frozen, or how refinement rollouts relate to final evaluation and the audit in Section 4.1. The latter relies on a single Claude Opus 5 judge without reported human calibration or inter-rater agreement. The authors appropriately qualify the findings as judge attributions, but 29 of 744 attempts containing attributed task defects and 16 outcome changes do not establish the prevalence of defective tasks or the judge's false-negative rate. The audit also does not assess whether tests accept implementations missing important specified behavior. Clearer process documentation and independent audit evidence are important because trustworthy grading is the benchmark's central contribution.

  2. The representativeness and robustness of the model comparisons need stronger support. Although Table 2 includes thirteen models, five are Claude variants and three are GPT variants, while Qwen and DeepSeek are absent. The evaluations of Kimi and GLM are useful, but additional versions from the two most represented families do not substitute for broader family coverage. The paper does not explain this selection, limiting the generality of its conclusions about model differentiation and reasoning behavior. Including one representative model from each omitted family, available at the time of evaluation, or explaining concrete evaluation constraints would help establish the scope of the findings.

    Reference-code overlap and statistical uncertainty further complicate the comparisons. Table 3(b) reports greater than 50% containment on 30, 25, and 24 tasks for Fable 5, Opus 5, and Opus 4.8, respectively, with maximum containment close to 98%. Comparing success rates between high-overlap tasks and the remaining tasks does not isolate the effect of memorization, since task difficulty and other properties may differ. A common-subset sensitivity analysis would help assess its influence on the comparisons. In addition, Table 2 reports four-run averages without uncertainty estimates; gaps such as 2.4 percentage points between Opus 5 and Fable 5, or 0.1 points between Opus 4.8 and GPT-5.6 Terra, should not be interpreted as stable ordering without paired analysis. The broad performance spread is informative, but finer comparisons require stronger support.

    A related issue concerns the comparison with NL2Repo-Bench: Appendix A reports strict repository-level pass@1, whereas the original benchmark's primary score is the average test pass rate. Using a common full-resolution metric is reasonable for the paper's stated objective, but a compressed distribution under that metric does not establish limited model discrimination under the original metric. Reporting both on the existing outputs would clarify the scope of the comparison.

Questions for authors
  1. Task refinement and independent validation: Which models and rollouts were used to refine tasks, what were the stopping criteria, and when was the benchmark frozen relative to the reported evaluations? Please clarify whether the Section 4.1 audit reused any refinement rollouts. What human or independent checks were performed to confirm that repairs removed unsupported constraints while preserving required functionality? A representative before/after task example would help make this process concrete.

  2. Interpretation of the solvability audit: How many distinct tasks account for the 29 attempts with attributed task defects, and how are the 16 outcome-changing cases distributed across models? Please report the relevant failed-attempt denominators and any available human checks of the judge's decisions, including checks of cases classified as model errors. This would help distinguish attempt-level outcome effects from task-level defect prevalence.

  3. Robustness of the model comparisons: Using the existing outputs, could you report model results on a common subset excluding tasks flagged for high reference-code containment by any evaluated model, together with uncertainty estimates that preserve task pairing and account for repeated runs within tasks? Please state how overlap flags from the single analyzed run are used for the four-run results. This is a sensitivity check, not a claim that low lexical overlap establishes absence of memorization.

  4. Metric and protocol alignment for NL2Repo-Bench: Could Appendix A also report the average test pass rate used as the primary score in the original NL2Repo-Bench paper? Its Table 3 reports 40.2% for Sonnet 4.5 with Claude Code, but only 3 fully resolved tasks out of 104; these are different metrics. Please clarify whether the claim of limited model differentiation is specifically about strict repository-level pass@1 under your scaffold and offline setup. What direct evidence separates specification–test defects from task difficulty or protocol effects as an explanation for the observed low full-resolution rates?

  5. Model-family coverage: What criteria guided model selection? Table 2 includes eight Claude/GPT variants but no Qwen or DeepSeek models. Could you include one representative model from each of these families that was available at evaluation time, or explain the specific constraints that prevented their inclusion? This would help assess how well the reported findings extend beyond the most heavily represented model families.

5 comments

Thank you for your valuable review. Please find our response below.

Task refinement details: We use Opus-4.6 and GPT-5.5 for rollout in task refinement. The rollout-guided review terminates when the number of fixed tasks per iteration falls below 3 (out of 186), which we believe is a reasonable threshold to upper-bound the evaluation noise.

The audit in section 4.1 uses a fresh collection of rollouts from the final dataset after the refinement loop completes. Refinement rollouts are not reused in the audit.

Interpretation of solvability audit: Please refer to Table 3 in our general response for the numbers. In addition, 26 distinct tasks account for the 29 attempts with attributed task defects. The authors manually reviewed a small sample of judge decisions to make sure the judge works as expected. Reviewing every single judge outcome is prohibitively expensive given the complexity of the tasks.

The audit also does not assess whether tests accept implementations missing important specified behavior.

Whether every specified behavior is exercised by the hidden tests is checked by the static review (Section 2.3), which iterates until no issue is found.

Model family coverage: Please find the result for DeepSeek-V4.1-Flash-1017 (max effort, averaged over 4 runs) below. We did not evaluate Qwen models because the open-sourced models only support 256K context natively which is too short for E2E-SWE. The 1M context versions are only available through API, which is difficult for us to access.

ModelPass@1 (%)Agent turnsOutput tokens (k)
DeepSeek-V4.1-Flash45.2173.7277.0

Robustness of model comparison: Please refer to Table 1 in our general response for uncertainty estimate of our main result (pass@1 averaged over 4 runs), and Table 2 for pass@1 after excluding tasks flagged for high reference-code containment by any evaluated model.

Metric and protocol alignment for NL2Repo-Bench: We believe pass@1 is a better metric than the fraction of tests passed for reasons stated in L256-263 of the paper. One piece of supporting evidence is that both Kimi and GLM have moved from reporting partial credit to binary score for ProgramBench in their model releases (from Kimi K3 to Kimi K3.1 Pro, and from GLM-5.2 to GLM-5.3).

Please see below for the average test pass rate of NL2RepoBench. In particular, the numbers for GPT-5.5, 5.6 Terra, and 5.6 Sol are clustered together, even though 5.6 Sol is expected to be better than the other two (same model family, different model generation and size).

ModelPass@1 (%)Avg. test pass rate (%)
Claude Fable 519.277.9
Claude Opus 522.176.5
Claude Sonnet 513.567.4
Claude Opus 4.813.574.7
Claude Opus 4.68.758.0
GPT-5.6 Sol7.760.7
GPT-5.6 Terra8.759.5
GPT-5.512.562.9
Muse Spark 1.39.659.2
Gemini 3.8 Flash8.757.2
Gemini 3.6 Flash8.755.7
Kimi K314.466.5
GLM-5.29.650.5

What direct evidence separates specification–test defects from task difficulty or protocol effects as an explanation for the observed low full-resolution rates?

Difficulty is unlikely the reason given the reference repo size comparison stated in L782-786. The protocol change from online to offline is a must because we observe that agents may directly clone the reference repo from Github when given Internet access, as mentioned in L759-761.

Thank you for the detailed response and the additional results. The four-run DeepSeek evaluation improves model-family coverage, and the fresh post-refinement audit addresses my concern about reusing refinement trajectories. The standard errors and the common-subset analysis are also useful: removing the 35 high-containment tasks preserves the top-five ordering in the reported run, which substantially alleviates my concern about the broad ranking being driven by reference-code overlap.

I have one remaining experimental request and two requests for clarification:

  1. Please add an evaluation of a representative Qwen model with 1M-context support. Qwen has officially released open-weight models with 1M-context support and local inference tools, so the stated API-only limitation does not adequately justify excluding the family. I request an evaluation of a representative Qwen model using a validated 1M-context configuration. Please use the same 186 tasks, agent scaffold, offline setting, and task budgets as the main evaluation, with four runs, and report pass@1 with standard error, average agent turns, and output tokens. Please also specify the exact checkpoint, serving framework, effective context limit, and any context-extension and decoding settings. This would directly address the remaining model-family coverage gap rather than only explaining its absence.

  2. Scope of the reliability claims. The stopping rule of fewer than three repaired tasks per iteration is a practical convergence criterion, not by itself an upper bound on residual evaluation noise. Similarly, potential bias from using the same model as solver and judge does not establish that the observed flip count is an upper bound. Please qualify these statements and report the size, selection procedure, and agreement results of the existing manual audit sample. This does not require manually reviewing every judgment.

  3. NL2Repo-Bench interpretation. Reporting average test pass rates addresses my request for the original metric. I agree with using strict pass@1 for your objective and with preventing reference-repository retrieval. However, reference LOC and an expected ordering among GPT variants do not independently establish that specification–test defects cause the observed scores. Please retain this as a possible explanation rather than treating task difficulty as ruled out.

Overall, the additional DeepSeek evaluation and robustness analyses have substantially strengthened the submission. The Qwen evaluation remains important to my assessment of model-family coverage. This request is about representative coverage and a well-specified evaluation, not about Qwen achieving any particular score or rank.

The four-run standard errors in Table 1 help quantify uncertainty in the reported mean scores, and the containment-filtered results in Table 2 provide a useful sensitivity check. Since the latter uses one run, the observed ranking stability should be interpreted within that scope; stability across repeated runs remains to be assessed.

Thank you for following up!

Additional Qwen result: We evaluated Qwen 3.8 Max, self-hosted with sglang at 1M context. We used medium effort because xhigh resulted in too long reasoning that exceeds the per-turn output budget (the same reason as described in L248-249). Please find the result below.

ModelPass@1 (%)Agent turnsOutput tokens (k)
Qwen3.8-Max36.0123.8234.8

Scope of the reliability claims: Your wording is more rigorous and we agree with your reframed statements. The manual audit was a qualitative check of the judge’s behavior. We randomly reviewed about 10 flagged issues from the entire pool, with each issue examined by one author.

NL2Repo-Bench interpretation: We agree with your point. In the paper, we do frame that as a possible explanation (L457), rather than a verified conclusion.

Reviewer_VjmyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces E2E-SWE, a benchmark for evaluating end-to-end repository-level code generation. The benchmark incorporates human-in-the-loop task construction, where human contributors create task specifications, test cases, and execution environments. The authors evaluate 13 frontier models on 186 tasks and report metrics including pass@1, the number of agent turns, and output token usage.

Critical strengths
  1. Repository-level code generation is an important and increasingly relevant task for evaluating the software engineering capabilities of foundation models and coding agents.

  2. The benchmark helps characterize the current capabilities and limitations of state-of-the-art models on repository-level software engineering tasks.

Critical weaknesses
  1. Novelty relative to existing repository-level benchmarks is unclear. There are already several benchmarks targeting repository-level code generation and software engineering, such as RepoCraft, RepoZero, and RepoGenesis. The paper would benefit from a clearer articulation of what fundamentally distinguishes E2E-SWE from these existing benchmarks.

  2. The benchmark construction process appears expensive and difficult to scale. E2E-SWE relies substantially on human effort to construct specifications, tests, and execution environments. While this may improve task quality, it also raises concerns about the scalability and reproducibility of the benchmark construction pipeline. With only 186 tasks, it would be helpful to understand the human effort required per task and whether the proposed methodology can realistically scale to thousands of tasks or be continuously refreshed as models improve.

  3. Reliability and coverage of test-based evaluation need stronger justification. The benchmark primarily relies on human-written tests to determine correctness. However, passing a limited set of tests does not necessarily imply that the generated implementation is semantically correct. Models or agents may exploit weaknesses in the tests, intentionally or unintentionally, producing solutions that pass the available tests without correctly implementing the intended functionality. The paper currently provides limited discussion of how the authors ensure sufficient test coverage, prevent test-specific overfitting or reward hacking, and verify consistency between specifications and tests.

  4. Evaluation is limited to a relatively simple agent. The experiments primarily evaluate models using a simple ReAct agent. However, repository-level coding performance can depend significantly on the agent harness, including context management, repository exploration, tool use, iterative debugging, and test execution. It would strengthen the evaluation to include representative state-of-the-art coding agents or harnesses, such as Claude Code or Codex coding agents. This would help distinguish whether failures are caused by the agent, or intrinsic task difficulty.

  5. The benchmark's insights could be developed further. Beyond ranking models according to pass@1, turns, and token usage, it is not yet clear what new insights E2E-SWE reveals about repository-level coding capabilities. A benchmark is particularly valuable when it enables analysis that cannot be easily obtained from existing benchmarks. For example, the authors could investigate failure modes, difficulty scaling, model-versus-agent bottlenecks, efficiency-quality tradeoffs, repository characteristics associated with failures, or capability differences across task categories.

Questions for authors
  1. What is the key conceptual or methodological novelty of E2E-SWE compared with existing repository-level code generation benchmarks such as RepoCraft, RepoZero, and RepoGenesis?

  2. How much human effort is required to construct and validate one E2E-SWE task, including specifications, tests, and execution environments? How would the proposed benchmark construction methodology scale from 186 tasks to thousands of tasks?

  3. How do the authors verify that the human-written tests provide sufficient coverage of the task specifications? Is there any measurement or manual auditing of test quality or coverage?

  4. Have the authors observed cases where agents pass tests by exploiting weaknesses in the test suite rather than correctly implementing the intended functionality? What safeguards are used to detect or prevent such behavior?

  5. How sensitive are the benchmark results to the choice of agent? Could the authors evaluate at least one or two stronger coding-agent harnesses to determine how much performance improvement comes from better agent?

  6. What new conclusions about frontier models' repository-level software engineering capabilities can be drawn from E2E-SWE that cannot already be observed from existing benchmarks? In my view, answering this question more clearly would substantially strengthen the paper's contribution.

1 comment

Thank you for your valuable review. Please find our response below.

Novelty w.r.t. existing benchmarks: RepoCraft, RepoZero, and RepoGenesis include 6, 35, and 30 repositories respectively for evaluation, compared to 186 by E2E-SWE which allows more robust measurement. More specifically,

  • RepoCraft only provides a short description as input but the grader checks up to hundreds of rubrics. Success depends heavily on prior knowledge of the reference codebases because the input requirement is thin while the repos are very famous (e.g., sklearn, pandas, sympy).

  • In RepoZero, each individual task only asks for implementing one API usage scenario, rather than a full repository that simultaneously satisfies all functional requirements.

  • RepoGenesis specifically targets web microservices. Every task is a REST service. In contrast, E2E-SWE covers diverse software domains. Complexity-wise, RepoGenesis reference implementations range from 100 to 2,426 lines of code, while E2E-SWE has a median of 7,360.

What is the key conceptual or methodological novelty of E2E-SWE compared with existing repository-level code generation benchmarks

We see the main contribution of a benchmark as enabling reliable measurement. Toward that end, E2E-SWE's methodological contribution is treating solvability as a first-class design goal. Specifically, specs are written after the hidden tests (Sec. 2.2) and refined by static and rollout-guided review (Sec. 2.3). This is what makes strict full-repository pass@1 usable.

Scalability: E2E-SWE is designed for evaluation. The current dataset of 186 tasks already provides reliable evaluation signals. Further scaling it up to thousands of tasks does not bring clear benefits and will make each evaluation run more expensive. Scaling up for training task curation is out of the scope of this paper.

Reliability and coverage of test-based evaluation: Below, we will address the concerns from three different perspectives.

  • Test coverage w.r.t. the original repo - We deliberately do not push the test coverage to be as high as possible, for reasons stated in L152-155. Our goal is to obtain a task that is hard enough to separate stronger models from weaker ones. We do not necessarily have to task the agent to reconstruct every piece of the reference repository. Covering a subset of the repo is fine as long as that still gives a hard task.

  • Test coverage w.r.t. spec - There are two gates for this. First, as described in Section 2.2, the task authors are instructed to draft the spec only after creating the tests. The spec is therefore determined by the test suite rather than the reference repo. Second, the static review in Section 2.3 examines spec-test correspondence to repair missing coverage.

  • Reward hacking - The agent cannot see the tests while building the repository.

Varying agent harness: Our main goal is to compare models, so we evaluate all models under the same simple scaffold. Nevertheless, we agree with the reviewer that harness sensitivity can be an interesting future work.

New insights about repository-level coding: We thank the reviewer for the suggestions on additional analysis. Given the page limit, we decided to prioritize the three analyses that we find most interesting (i.e., generated repo size in Section 4.2, reasoning pattern in 4.3, and memorization behavior in 4.4). To the best of our knowledge, there are no similar studies from existing repository-generation benchmark works.

More importantly, our benchmark demonstrates that frontier models can already build complete repositories from well-written specifications, evidenced by significant pass@1 scores which were rarely observed from prior repository-generation benchmarks with the same level of task complexity. This observation would not be possible without our rigorous task construction process that emphasizes task solvability. Furthermore, E2E-SWE has stronger discriminative power than previous benchmarks, such as NL2Repo-Bench, in separating models by their whole-repository generation capability.

Reviewer_VUNoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces E2E-SWE, 186 from-scratch whole-repository generation tasks across 11 languages, with specifications and hidden tests newly written by engineers working with an LLM and repeatedly repaired through static and rollout audits. Across 13 frontier models, pass@1 ranges from 11.7% to 67.7%, giving clear separation.

Critical strengths
  1. Tests and specifications are written from scratch and check only externally observable behavior, which avoids the implementation coupling that comes with reusing the original repositories' tests.
  2. The evaluation pipeline is rigorous: agents build offline, grading installs the repository in a fresh container, and the paper also reports a 12-gram memorization analysis.
Critical weaknesses
  1. In Table 3(b), Fable 5, Opus 5, and Opus 4.8 exceed 50% overlap with the reference code on 24–30 tasks, all three rank near the top, and Opus 5 resolves 9.6 pp more of those tasks. How do the authors rule out that the rankings are inflated by memorization?
  2. Specifications are repaired and tests relaxed based on failures in model rollouts, yet the paper doesn't say which models were used, and Section 4.1 then uses Opus 5 as the judge. How do you ensure that neither the repairs nor the judging favor particular models, so that the 2.2% defect rate can serve as independent evidence?
Questions for authors
  1. Which models were used in the rollout audits during task construction? If the audit is rerun with models that took no part in the repairs, are the defect rate and the rankings stable?
  2. After removing tasks with more than 50% 12-gram overlap with the reference code, how do the pass@1 scores and rankings of the 13 models change?
  3. How much does pass@1 vary across the 4 independent runs? Are gaps between adjacent models, such as Opus 5 and Fable 5, statistically significant?
3 comments

Thank you for your valuable review. Please find our response below.

Impact of memorization: We would like to highlight that memorization can happen to almost all public benchmarks based on OSS Github repositories, considering how pre-training data is sourced. It is not unique to E2E-SWE. Our memorization study is meant to provide a rough estimate of the impact. Filtering the tasks based on the 12-gram containment would make the benchmark model-dependent, hurting its value for evaluating new models in the future.

As a probe, please refer to Table 2 in our general response for pass@1 and ranking before and after removing tasks where any model achieves >50% containment. Specifically, the ranking largely remains unchanged, except for the four models whose pass@1 clusters between 42-45.

Task repair details: Task repair was guided by the failures of two rollout models, Opus 4.6 and GPT-5.5. Table 3 in our general response shows the breakdown of the audit result in Section 4.1.

  • GPT-5.6 Sol and Kimi K3 took no part in the repairs and are not the judge, so their counts are the cleanest evidence: 3 and 4 outcome flips over 186 tasks (1.6% and 2.2%).
  • Opus 4.6 has 0 flips since it was used to repair the tasks.
  • The Opus 5 count is likely inflated because Opus 5 is also the judge. If Opus 5 as the solver overlooks a detail in the spec or fails to derive an implicit requirement from the spec, Opus 5 as the judge can overlook the same. The judge then concludes that the specification does not determine the tested behavior and attributes the solver's error to the task. This shared blind spot leads the judge to blame the task, not the model, so the 9 flips (4.8%) for Opus 5 should be read as an upper bound.

Main result variance: Please refer to Table 1 in our general response for standard error of the 4-run pass@1.

I appreciate the author’s hard work, which has gone some way toward alleviating my concerns. My one remaining reservation is that I believe a two-way alignment between tests and specifications could lead to two issues: 1. Adjusting tests to conform to specifications might result in tests that are too lenient and fail to cover complex, end-to-end behaviors. 2. Modifying tests based on specifications could cause the specifications themselves to effectively devolve into nothing more than a list of test outcomes. How does the author address or avoid these two problems?

Thank you for following up!

  1. Adjusting tests to conform to specifications might result in tests that are too lenient and fail to cover complex, end-to-end behaviors.

During initial task authoring, tests are written before the spec. We design the spec to be test-driven instead of the other way around, to maximize the difficulty of the tests.

During quality refinement, we ask the agent to do minimal modification when fixing issues. That typically involves changing how functions and classes are called and how the final outcomes are asserted, instead of repurposing or removing an entire end-to-end test.

  1. Modifying tests based on specifications could cause the specifications themselves to effectively devolve into nothing more than a list of test outcomes.

You probably mean modifying spec based on tests. During initial task authoring, we explicitly instructed authors to avoid leaking either implementation details or hidden test outcomes. The spec should focus on describing the behaviors. During quality refinement, the same rule is applied in the prompt, and we include a self-review stage to rework any fix that leaks test outcome or implementation detail.

Overall, the effectiveness of our design is reflected by the evaluation results. Both issues you mentioned would make tasks over easy, thus losing discriminative power and becoming saturated by the strongest models. Empirically, E2E-SWE is challenging enough to provide clear model differentiation and reserves enough headroom for the best models, as shown in Table 2 of the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers

We thank all the reviewers for their insightful comments. In this general response, we summarize some common concerns raised by reviewers and how we address them, followed by additional results to support our claims.

Uncertainty estimate of the main results: Reviewer VUNo and Bkzy both raised questions about the variance of the pass@1 reported in our main result table. We augment the reported pass@1 with standard error in Table 1 below to provide a clear picture of performance comparison.

Table 1. Pass@1 as mean ± standard error over 4 independent runs

ModelPass@1
Claude Fable 565.3 ± 0.6
Claude Opus 567.7 ± 1.0
Claude Sonnet 543.1 ± 0.7
Claude Opus 4.851.9 ± 1.8
Claude Opus 4.622.6 ± 0.9
GPT-5.6 Sol61.2 ± 0.8
GPT-5.6 Terra51.8 ± 1.4
GPT-5.546.1 ± 0.8
Muse Spark 1.345.2 ± 0.8
Gemini 3.8 Flash36.7 ± 1.3
Gemini 3.6 Flash11.7 ± 1.0
Kimi K342.3 ± 1.8
GLM-5.222.2 ± 0.9

 

Scores after excluding memorized tasks: Reviewer VUNo and Bkzy both asked about the impact of memorization on final scores. In Table 2 below, we compare pass@1 before and after removing the 35 tasks where at least one model’s implementation has more than 50% 12-gram containment w.r.t. the reference repo. Performance rankings are largely preserved except for the four models whose pass@1 are very close, in which case any random perturbation may change the ranking.

Table 2. Pass@1 after removing the 35 tasks where any model exceeds 50% 12-gram containment (1 run)

ModelPass@1 (all 186 tasks)RankPass@1 (151 kept tasks)RankΔ (pts)
Claude Opus 567.7165.61-2.2
Claude Fable 564.0264.22+0.3
GPT-5.6 Sol59.1359.63+0.5
Claude Opus 4.854.8454.34-0.5
GPT-5.6 Terra51.1551.75+0.6
GPT-5.545.2642.49-2.8
Muse Spark 1.344.6745.06+0.4
Kimi K343.5845.06+1.5
Claude Sonnet 542.5944.48+1.9
Gemini 3.8 Flash33.31033.810+0.4
GLM-5.223.71126.511+2.8
Claude Opus 4.622.01221.912-0.2
Gemini 3.6 Flash14.01313.213-0.7

 

Details about rollout review and fairness audit Reviewer VUNo and Bkzy both asked for more details about how the rollout-guided review was conducted during task construction and how that overlaps with the fairness audit in Section 4.1. We use Opus-4.6 and GPT-5.5 for rollout in quality refinement during task construction. Table 3 below shows the model-level breakdown of fairness audit of the final dataset from Section 4.1, including the number of rollout attempts with at least one test failure and the outcome flips per model. Specifically, GPT-5.6 Sol and Kimi K3, which were not used for task repair nor as the judge, both show a small number of outcome flips due to fairness issues. Opus-5’s outcome flips are likely inflated because the same model is also used as the judge. In this case, the judge may overlook the same detail that is derivable from the spec but missed by the solver, and hence attribute the test failure to the task instead of to the model. We read that number as an upper bound.

Table 3. Task fairness audit: failed attempts and outcome flips (1 run)

ModelFailed attemptsOutcome flips
Claude Opus 5609
GPT-5.6 Sol763
Kimi K31054
Claude Opus 4.61450
Total38616
Area_Chair_mxadOpenReview ↗

Please engage in the discussion

Dear Reviewers,

Thank you for your thoughtful reviews. The authors have posted their rebuttal, please read it and share any updated assessments.

As you know, we have only 3 reviews in total. Each piece of your opinions is valuable and may have crucial impact on the final decison. So please take it seriously.

Thanks again for your contribution!

Best Regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.