- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-Bench-XL, a benchmark of 100 software engineering tasks collected from four large repositories: VS Code, Elasticsearch, TypeScript, and Transformers. The benchmark combines synthesized problem statements, generated test oracles, and containerized environments in Harbor format. Its median repository size is 44.9 million source-code tokens.The authors evaluate five models under OpenHands, with five attempts per task, reporting pass@1 between 22.5% and 47.6%. A comparison using Claude Sonnet 5 across four benchmarks finds more searches and broader directory exploration on SWE-Bench-XL. The paper interprets these behaviors as evidence that the benchmark places greater demands on repository navigation.
Critical strengths
The benchmark covers repositories ranging from 14.8 million to 148.2 million tokens and complements evaluations centered on smaller codebases. Its focus on localized repairs within large repositories offers a useful setting for studying code discovery and context selection. The experiments use a common agent framework and repeated attempts, while reporting repository-specific performance and trajectory statistics. The analysis of patch size and dependencies adds useful context beyond an aggregate leaderboard, and Harbor packaging offers a practical route to reuse.
Critical weaknesses
1.Section 2.4 primarily describes checking that tests fail before the fix and pass with the reference patch. This does not establish that synthesized requirements are complete or that plausible incorrect patches are rejected. No independent audit or mutation-testing results are reported. Prior work such as UTBoost demonstrates why this distinction matters. ACL Anthology
2.Table 4 compares different repositories, languages, task distributions, and problem-generation procedures. Using the same model and harness does not isolate repository size. More searches and directory visits therefore provide descriptive evidence, but cannot establish that the benchmark specifically measures stronger navigation demands.
3.The higher Ops/|G| and Nav/W ratios partly reflect smaller reference patches and fewer writes, rather than necessarily greater navigation difficulty. Likewise, counting all tokens in directly imported files does not establish how much context is actually needed to solve a task.
4.Section 3.2 excludes error and missing outcomes without reporting their frequency, causes, or distribution across models and repositories. Excluding difficult runs or agent-induced failures could change both pass@1 and trajectory averages. The treatment of tasks with no valid attempts is also unclear.
5.Tables 3 and 4 provide no confidence intervals. Section 4.2 nevertheless states that differences are not statistically significant without identifying the test or reporting its results. Moreover, failure to detect a difference does not establish equivalent difficulty or trajectory length.
6.The paper does not specify the construction model, generation prompts, candidate rejection rates, or exact OpenHands configuration. Actual context limits, time budgets, sampling settings, and hardware resources are also missing, limiting reproducibility and interpretation of an evaluation centered on context constraints.
Questions for authors
1.What validation beyond reference-patch success and base-commit failure was performed? Please report any independent review of problem statements and tests, or checks against plausible incorrect patches and unintended solution hints.
2.How many attempts were excluded for each model and repository, and for what reasons? How are timeouts, build failures, and tasks with no valid attempts treated? Would the conclusions change under an alternative treatment of unresolved outcomes?
3.Can you provide a diagnostic comparison with the relevant source files identified for the agent? This would help distinguish failures in locating code from failures in implementing the fix and directly inform the benchmark’s claimed navigation focus.
4.What statistical procedure supports the nonsignificance statement in Section 4.2? Please provide uncertainty estimates alongside the exact generation and evaluation configurations needed to reproduce the results.
2 comments
Q3. "Can you provide a diagnostic comparison with the relevant source files identified for the agent? This would help distinguish failures in locating code from failures in implementing the fix […]"
We ran this comparison with OpenHands and GPT-5.6 Terra. We added the list of source files changed by the reference patch to the problem statement, without the actual patch. We tried three different instructions which nudge the model to 1) look carefully at these files, 2) edit these files, and 3) edit these files after exploring the codebase. Without the list, pass@1 is 27.1%. With the list, it is 27.6%, 25.6% and 29.0%, respectively. These differences are smaller than the confidence interval of the baseline (± 6.3 points).
For this model, supplying the list of files to edit does not make the tasks easier. We believe the conclusions that can be drawn from this ablation are limited, however. While telling the agent where to edit allows the agent to focus on that part of the codebase earlier on in the trajectory, the agent still has to find and read the code that these files depend on to write a correct fix that is coherent with the codebase.
We thank the reviewer for their time and constructive feedback.
W1/Q1. "What validation beyond reference-patch success and base-commit failure was performed? Please report any independent review of problem statements and tests, or checks against plausible incorrect patches and unintended solution hints."
We have added Appendix B describing our data generation pipeline. Please also see "Task construction and validation" in the general response.
W2. "Using the same model and harness does not isolate repository size. More searches and directory visits therefore provide descriptive evidence, but cannot establish that the benchmark specifically measures stronger navigation demands."
We agree. The benchmarks differ in repositories, languages and tasks, not only in size, so the comparison in Table 4 does not isolate the effect of repository size. We report the searches and directory visits to describe how the same agent behaves in SWE-Bench-XL and in existing benchmarks. We do not claim that they establish stronger navigation demands, and we have revised the wording of the abstract and Section 4.2 accordingly.
W3. "The higher Ops/|G| and Nav/W ratios partly reflect smaller reference patches and fewer writes, rather than necessarily greater navigation difficulty. Likewise, counting all tokens in directly imported files does not establish how much context is actually needed to solve a task."
We agree that both ratios depend on the size of the reference patch and on the number of writes, so they do not measure navigation difficulty. We report them to describe the composition of the work. In SWE-Bench-XL the agent makes 7.1 navigation operations per write, against 4.8 in SWE-Bench-Verified, 4.3 in SWE-Bench-Pro and 3.4 in DeepSWE, and it uses a similar number of operations as in DeepSWE for a change that touches fewer files. Navigation is thus a larger share of an attempt in SWE-Bench-XL. The tokens in directly imported files measure the code that the edited files depend on, which is an upper bound on the context a solution needs. We have revised Sections 2 and 4.2 to state both limits.
W4/Q2. "How many attempts were excluded for each model and repository, and for what reasons? How are timeouts, build failures, and tasks with no valid attempts treated? Would the conclusions change under an alternative treatment of unresolved outcomes?"
We have added Appendix D which reports, for each harness and model, the number of attempts that reached the time limit, stopped with another type of error, or that have no recorded result. Over the 5,000 attempts on SWE-Bench-XL, 123 reached the 60-minute limit of OpenHands, 4 stopped with another error and 27 have no recorded result. There is no separate category for build failures.
The paper excludes these attempts from pass@1. Appendix D also provides pass@1 with errors counted as failures. With OpenHands, pass@1 decreases by 4.8 points for GPT-5.6 Sol, by 2.4 points for Claude Opus 5 and by at most 0.5 points for the other models. The order of the models does not change.
W5/Q4. "Tables 3 and 4 provide no confidence intervals. […] Moreover, failure to detect a difference does not establish equivalent difficulty or trajectory length. […] What statistical procedure supports the nonsignificance statement in Section 4.2?"
We added 95% confidence intervals to all results in Tables 3 and 4. We calculate them from 10,000 bootstrap resamples of the tasks (Section 3.2). The tables now include the confidence intervals.
We replaced the nonsignificant statement in Section 4.2 with a statement based on these intervals. For pass@1, operations and reads, the intervals of SWE-Bench-XL and DeepSWE overlap. Each estimate is inside the interval of the other benchmark (pass@1: 22.5% ± 6.4 and 25.3% ± 5.6).
We changed "nearly equal difficulty and total trajectory size" to "similar pass@1 and trajectory size". Section 4.2 now also reports the results with a second harness, OpenCode. pass@1 is also similar on the two benchmarks, but DeepSWE trajectories are longer (93.8 operations for each attempt, against 75.3 on SWE-Bench-XL).
W6. "The paper does not specify the construction model, generation prompts, candidate rejection rates, or exact OpenHands configuration."
We have added Appendix B, which describes the construction pipeline and its agents.
We have added Appendix C, which provides the settings of both harnesses.