- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
E2E-SWE evaluates whether coding agents can construct complete, installable repositories from natural-language specifications in an empty workspace. The benchmark contains 186 tasks spanning 11 programming languages, grounded in public GitHub repositories. Software engineers collaborate with LLMs to author public-interface hidden tests and corresponding specifications, followed by static audits and rollout-guided task refinement. Agents work offline, and their generated repositories are installed and graded in fresh containers. The authors evaluate 13 models over four independent full-corpus runs per model, reporting pass@1 from 11.7% to 67.7%. Additional analyses examine reasoning effort, generated code size, the distribution of reasoning across turns, residual specification–test defects, and overlap with reference code.
Critical strengths
-
An important evaluation target with a well-motivated approach to test fairness. Whole-repository generation makes specification–test mismatches particularly consequential because a single unsupported assertion can fail an otherwise correct implementation. Authoring tests around public behavior and aligning the specification with those tests directly addresses this problem, while allowing agents to choose their internal architecture. This gives the benchmark a useful focus on implementing a stated behavioral contract.
-
A substantial evaluation effort with a sensible execution protocol. The benchmark covers 186 tasks across 11 languages and evaluates 13 models with four independent runs per model. Offline generation and installation in a fresh grading container reduce dependence on incidental development state. The wide observed performance range, reasoning-effort experiments, and explicit reporting of residual task issues and reference-code overlap provide useful evidence about current repository-generation systems.
Critical weaknesses
-
The central claim of reliable specification–test alignment needs stronger independent validation. Section 2.3 does not specify the models and stopping criteria used in refinement, when tasks were frozen, or how refinement rollouts relate to final evaluation and the audit in Section 4.1. The latter relies on a single Claude Opus 5 judge without reported human calibration or inter-rater agreement. The authors appropriately qualify the findings as judge attributions, but 29 of 744 attempts containing attributed task defects and 16 outcome changes do not establish the prevalence of defective tasks or the judge's false-negative rate. The audit also does not assess whether tests accept implementations missing important specified behavior. Clearer process documentation and independent audit evidence are important because trustworthy grading is the benchmark's central contribution.
-
The representativeness and robustness of the model comparisons need stronger support. Although Table 2 includes thirteen models, five are Claude variants and three are GPT variants, while Qwen and DeepSeek are absent. The evaluations of Kimi and GLM are useful, but additional versions from the two most represented families do not substitute for broader family coverage. The paper does not explain this selection, limiting the generality of its conclusions about model differentiation and reasoning behavior. Including one representative model from each omitted family, available at the time of evaluation, or explaining concrete evaluation constraints would help establish the scope of the findings.
Reference-code overlap and statistical uncertainty further complicate the comparisons. Table 3(b) reports greater than 50% containment on 30, 25, and 24 tasks for Fable 5, Opus 5, and Opus 4.8, respectively, with maximum containment close to 98%. Comparing success rates between high-overlap tasks and the remaining tasks does not isolate the effect of memorization, since task difficulty and other properties may differ. A common-subset sensitivity analysis would help assess its influence on the comparisons. In addition, Table 2 reports four-run averages without uncertainty estimates; gaps such as 2.4 percentage points between Opus 5 and Fable 5, or 0.1 points between Opus 4.8 and GPT-5.6 Terra, should not be interpreted as stable ordering without paired analysis. The broad performance spread is informative, but finer comparisons require stronger support.
A related issue concerns the comparison with NL2Repo-Bench: Appendix A reports strict repository-level pass@1, whereas the original benchmark's primary score is the average test pass rate. Using a common full-resolution metric is reasonable for the paper's stated objective, but a compressed distribution under that metric does not establish limited model discrimination under the original metric. Reporting both on the existing outputs would clarify the scope of the comparison.
Questions for authors
-
Task refinement and independent validation: Which models and rollouts were used to refine tasks, what were the stopping criteria, and when was the benchmark frozen relative to the reported evaluations? Please clarify whether the Section 4.1 audit reused any refinement rollouts. What human or independent checks were performed to confirm that repairs removed unsupported constraints while preserving required functionality? A representative before/after task example would help make this process concrete.
-
Interpretation of the solvability audit: How many distinct tasks account for the 29 attempts with attributed task defects, and how are the 16 outcome-changing cases distributed across models? Please report the relevant failed-attempt denominators and any available human checks of the judge's decisions, including checks of cases classified as model errors. This would help distinguish attempt-level outcome effects from task-level defect prevalence.
-
Robustness of the model comparisons: Using the existing outputs, could you report model results on a common subset excluding tasks flagged for high reference-code containment by any evaluated model, together with uncertainty estimates that preserve task pairing and account for repeated runs within tasks? Please state how overlap flags from the single analyzed run are used for the four-run results. This is a sensitivity check, not a claim that low lexical overlap establishes absence of memorization.
-
Metric and protocol alignment for NL2Repo-Bench: Could Appendix A also report the average test pass rate used as the primary score in the original NL2Repo-Bench paper? Its Table 3 reports 40.2% for Sonnet 4.5 with Claude Code, but only 3 fully resolved tasks out of 104; these are different metrics. Please clarify whether the claim of limited model differentiation is specifically about strict repository-level pass@1 under your scaffold and offline setup. What direct evidence separates specification–test defects from task difficulty or protocol effects as an explanation for the observed low full-resolution rates?
-
Model-family coverage: What criteria guided model selection? Table 2 includes eight Claude/GPT variants but no Qwen or DeepSeek models. Could you include one representative model from each of these families that was available at evaluation time, or explain the specific constraints that prevented their inclusion? This would help assess how well the reported findings extend beyond the most heavily represented model families.
5 comments
Thank you for the detailed response and the additional results. The four-run DeepSeek evaluation improves model-family coverage, and the fresh post-refinement audit addresses my concern about reusing refinement trajectories. The standard errors and the common-subset analysis are also useful: removing the 35 high-containment tasks preserves the top-five ordering in the reported run, which substantially alleviates my concern about the broad ranking being driven by reference-code overlap.
I have one remaining experimental request and two requests for clarification:
-
Please add an evaluation of a representative Qwen model with 1M-context support. Qwen has officially released open-weight models with 1M-context support and local inference tools, so the stated API-only limitation does not adequately justify excluding the family. I request an evaluation of a representative Qwen model using a validated 1M-context configuration. Please use the same 186 tasks, agent scaffold, offline setting, and task budgets as the main evaluation, with four runs, and report pass@1 with standard error, average agent turns, and output tokens. Please also specify the exact checkpoint, serving framework, effective context limit, and any context-extension and decoding settings. This would directly address the remaining model-family coverage gap rather than only explaining its absence.
-
Scope of the reliability claims. The stopping rule of fewer than three repaired tasks per iteration is a practical convergence criterion, not by itself an upper bound on residual evaluation noise. Similarly, potential bias from using the same model as solver and judge does not establish that the observed flip count is an upper bound. Please qualify these statements and report the size, selection procedure, and agreement results of the existing manual audit sample. This does not require manually reviewing every judgment.
-
NL2Repo-Bench interpretation. Reporting average test pass rates addresses my request for the original metric. I agree with using strict pass@1 for your objective and with preventing reference-repository retrieval. However, reference LOC and an expected ordering among GPT variants do not independently establish that specification–test defects cause the observed scores. Please retain this as a possible explanation rather than treating task difficulty as ruled out.
Overall, the additional DeepSeek evaluation and robustness analyses have substantially strengthened the submission. The Qwen evaluation remains important to my assessment of model-family coverage. This request is about representative coverage and a well-specified evaluation, not about Qwen achieving any particular score or rank.
The four-run standard errors in Table 1 help quantify uncertainty in the reported mean scores, and the containment-filtered results in Table 2 provide a useful sensitivity check. Since the latter uses one run, the observed ranking stability should be interpreted within that scope; stability across repeated runs remains to be assessed.
Thank you for following up!
Additional Qwen result: We evaluated Qwen 3.8 Max, self-hosted with sglang at 1M context. We used medium effort because xhigh resulted in too long reasoning that exceeds the per-turn output budget (the same reason as described in L248-249). Please find the result below.
| Model | Pass@1 (%) | Agent turns | Output tokens (k) |
|---|---|---|---|
| Qwen3.8-Max | 36.0 | 123.8 | 234.8 |
Scope of the reliability claims: Your wording is more rigorous and we agree with your reframed statements. The manual audit was a qualitative check of the judge’s behavior. We randomly reviewed about 10 flagged issues from the entire pool, with each issue examined by one author.
NL2Repo-Bench interpretation: We agree with your point. In the paper, we do frame that as a possible explanation (L457), rather than a verified conclusion.
Thank you for your valuable review. Please find our response below.
Task refinement details: We use Opus-4.6 and GPT-5.5 for rollout in task refinement. The rollout-guided review terminates when the number of fixed tasks per iteration falls below 3 (out of 186), which we believe is a reasonable threshold to upper-bound the evaluation noise.
The audit in section 4.1 uses a fresh collection of rollouts from the final dataset after the refinement loop completes. Refinement rollouts are not reused in the audit.
Interpretation of solvability audit: Please refer to Table 3 in our general response for the numbers. In addition, 26 distinct tasks account for the 29 attempts with attributed task defects. The authors manually reviewed a small sample of judge decisions to make sure the judge works as expected. Reviewing every single judge outcome is prohibitively expensive given the complexity of the tasks.
Whether every specified behavior is exercised by the hidden tests is checked by the static review (Section 2.3), which iterates until no issue is found.
Model family coverage: Please find the result for DeepSeek-V4.1-Flash-1017 (max effort, averaged over 4 runs) below. We did not evaluate Qwen models because the open-sourced models only support 256K context natively which is too short for E2E-SWE. The 1M context versions are only available through API, which is difficult for us to access.
Robustness of model comparison: Please refer to Table 1 in our general response for uncertainty estimate of our main result (pass@1 averaged over 4 runs), and Table 2 for pass@1 after excluding tasks flagged for high reference-code containment by any evaluated model.
Metric and protocol alignment for NL2Repo-Bench: We believe pass@1 is a better metric than the fraction of tests passed for reasons stated in L256-263 of the paper. One piece of supporting evidence is that both Kimi and GLM have moved from reporting partial credit to binary score for ProgramBench in their model releases (from Kimi K3 to Kimi K3.1 Pro, and from GLM-5.2 to GLM-5.3).
Please see below for the average test pass rate of NL2RepoBench. In particular, the numbers for GPT-5.5, 5.6 Terra, and 5.6 Sol are clustered together, even though 5.6 Sol is expected to be better than the other two (same model family, different model generation and size).
Difficulty is unlikely the reason given the reference repo size comparison stated in L782-786. The protocol change from online to offline is a must because we observe that agents may directly clone the reference repo from Github when given Internet access, as mentioned in L759-761.