acceptodds
Under review as a conference paper at ICLR 2027

SWE-Bench-XL: Evaluating Coding Agents on Very Large Repositories

12 views

Abstract

Coding agents are evaluated on benchmarks of repository-level tasks mined from real pull requests, but current reference benchmarks mostly draw from small codebases. Real industrial and open-source development happens in much larger repositories, where an agent must navigate the code and manage a limited context window. We close this gap with SWE-Bench-XL, a benchmark of 100 repro- ducible tasks mined from four of the largest open-source repositories on GitHub, with a mean per-instance source-code size roughly an order of magnitude larger than reference benchmarks. Across five frontier models evaluated under Open- Hands, pass@1 ranges from 22.5% to 47.6%. Trajectory analysis shows that agents make more codebase searches and visit more directories than in any other benchmark, indicating that SWE-Bench-XL better exercises the capabilities of agents in navigating codebases. We release SWE-Bench-XL as reproducible Har- bor tasks.

Primary area:
datasets and benchmarks
Keywords:
coding agents, benchmarks, software engineering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_AZ2LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SWE-Bench-XL, a benchmark of 100 software engineering tasks collected from four large repositories: VS Code, Elasticsearch, TypeScript, and Transformers. The benchmark combines synthesized problem statements, generated test oracles, and containerized environments in Harbor format. Its median repository size is 44.9 million source-code tokens.The authors evaluate five models under OpenHands, with five attempts per task, reporting pass@1 between 22.5% and 47.6%. A comparison using Claude Sonnet 5 across four benchmarks finds more searches and broader directory exploration on SWE-Bench-XL. The paper interprets these behaviors as evidence that the benchmark places greater demands on repository navigation.

Critical strengths

The benchmark covers repositories ranging from 14.8 million to 148.2 million tokens and complements evaluations centered on smaller codebases. Its focus on localized repairs within large repositories offers a useful setting for studying code discovery and context selection. The experiments use a common agent framework and repeated attempts, while reporting repository-specific performance and trajectory statistics. The analysis of patch size and dependencies adds useful context beyond an aggregate leaderboard, and Harbor packaging offers a practical route to reuse.

Critical weaknesses

1.Section 2.4 primarily describes checking that tests fail before the fix and pass with the reference patch. This does not establish that synthesized requirements are complete or that plausible incorrect patches are rejected. No independent audit or mutation-testing results are reported. Prior work such as UTBoost demonstrates why this distinction matters. ACL Anthology

2.Table 4 compares different repositories, languages, task distributions, and problem-generation procedures. Using the same model and harness does not isolate repository size. More searches and directory visits therefore provide descriptive evidence, but cannot establish that the benchmark specifically measures stronger navigation demands.

3.The higher Ops/|G| and Nav/W ratios partly reflect smaller reference patches and fewer writes, rather than necessarily greater navigation difficulty. Likewise, counting all tokens in directly imported files does not establish how much context is actually needed to solve a task.

4.Section 3.2 excludes error and missing outcomes without reporting their frequency, causes, or distribution across models and repositories. Excluding difficult runs or agent-induced failures could change both pass@1 and trajectory averages. The treatment of tasks with no valid attempts is also unclear.

5.Tables 3 and 4 provide no confidence intervals. Section 4.2 nevertheless states that differences are not statistically significant without identifying the test or reporting its results. Moreover, failure to detect a difference does not establish equivalent difficulty or trajectory length.

6.The paper does not specify the construction model, generation prompts, candidate rejection rates, or exact OpenHands configuration. Actual context limits, time budgets, sampling settings, and hardware resources are also missing, limiting reproducibility and interpretation of an evaluation centered on context constraints.

Questions for authors

1.What validation beyond reference-patch success and base-commit failure was performed? Please report any independent review of problem statements and tests, or checks against plausible incorrect patches and unintended solution hints.

2.How many attempts were excluded for each model and repository, and for what reasons? How are timeouts, build failures, and tasks with no valid attempts treated? Would the conclusions change under an alternative treatment of unresolved outcomes?

3.Can you provide a diagnostic comparison with the relevant source files identified for the agent? This would help distinguish failures in locating code from failures in implementing the fix and directly inform the benchmark’s claimed navigation focus.

4.What statistical procedure supports the nonsignificance statement in Section 4.2? Please provide uncertainty estimates alongside the exact generation and evaluation configurations needed to reproduce the results.

2 comments

We thank the reviewer for their time and constructive feedback.

W1/Q1. "What validation beyond reference-patch success and base-commit failure was performed? Please report any independent review of problem statements and tests, or checks against plausible incorrect patches and unintended solution hints."

We have added Appendix B describing our data generation pipeline. Please also see "Task construction and validation" in the general response.

W2. "Using the same model and harness does not isolate repository size. More searches and directory visits therefore provide descriptive evidence, but cannot establish that the benchmark specifically measures stronger navigation demands."

We agree. The benchmarks differ in repositories, languages and tasks, not only in size, so the comparison in Table 4 does not isolate the effect of repository size. We report the searches and directory visits to describe how the same agent behaves in SWE-Bench-XL and in existing benchmarks. We do not claim that they establish stronger navigation demands, and we have revised the wording of the abstract and Section 4.2 accordingly.

W3. "The higher Ops/|G| and Nav/W ratios partly reflect smaller reference patches and fewer writes, rather than necessarily greater navigation difficulty. Likewise, counting all tokens in directly imported files does not establish how much context is actually needed to solve a task."

We agree that both ratios depend on the size of the reference patch and on the number of writes, so they do not measure navigation difficulty. We report them to describe the composition of the work. In SWE-Bench-XL the agent makes 7.1 navigation operations per write, against 4.8 in SWE-Bench-Verified, 4.3 in SWE-Bench-Pro and 3.4 in DeepSWE, and it uses a similar number of operations as in DeepSWE for a change that touches fewer files. Navigation is thus a larger share of an attempt in SWE-Bench-XL. The tokens in directly imported files measure the code that the edited files depend on, which is an upper bound on the context a solution needs. We have revised Sections 2 and 4.2 to state both limits.

W4/Q2. "How many attempts were excluded for each model and repository, and for what reasons? How are timeouts, build failures, and tasks with no valid attempts treated? Would the conclusions change under an alternative treatment of unresolved outcomes?"

We have added Appendix D which reports, for each harness and model, the number of attempts that reached the time limit, stopped with another type of error, or that have no recorded result. Over the 5,000 attempts on SWE-Bench-XL, 123 reached the 60-minute limit of OpenHands, 4 stopped with another error and 27 have no recorded result. There is no separate category for build failures.

The paper excludes these attempts from pass@1. Appendix D also provides pass@1 with errors counted as failures. With OpenHands, pass@1 decreases by 4.8 points for GPT-5.6 Sol, by 2.4 points for Claude Opus 5 and by at most 0.5 points for the other models. The order of the models does not change.

W5/Q4. "Tables 3 and 4 provide no confidence intervals. […] Moreover, failure to detect a difference does not establish equivalent difficulty or trajectory length. […] What statistical procedure supports the nonsignificance statement in Section 4.2?"

We added 95% confidence intervals to all results in Tables 3 and 4. We calculate them from 10,000 bootstrap resamples of the tasks (Section 3.2). The tables now include the confidence intervals.

We replaced the nonsignificant statement in Section 4.2 with a statement based on these intervals. For pass@1, operations and reads, the intervals of SWE-Bench-XL and DeepSWE overlap. Each estimate is inside the interval of the other benchmark (pass@1: 22.5% ± 6.4 and 25.3% ± 5.6).

We changed "nearly equal difficulty and total trajectory size" to "similar pass@1 and trajectory size". Section 4.2 now also reports the results with a second harness, OpenCode. pass@1 is also similar on the two benchmarks, but DeepSWE trajectories are longer (93.8 operations for each attempt, against 75.3 on SWE-Bench-XL).

W6. "The paper does not specify the construction model, generation prompts, candidate rejection rates, or exact OpenHands configuration."

We have added Appendix B, which describes the construction pipeline and its agents.

We have added Appendix C, which provides the settings of both harnesses.

Q3. "Can you provide a diagnostic comparison with the relevant source files identified for the agent? This would help distinguish failures in locating code from failures in implementing the fix […]"

We ran this comparison with OpenHands and GPT-5.6 Terra. We added the list of source files changed by the reference patch to the problem statement, without the actual patch. We tried three different instructions which nudge the model to 1) look carefully at these files, 2) edit these files, and 3) edit these files after exploring the codebase. Without the list, pass@1 is 27.1%. With the list, it is 27.6%, 25.6% and 29.0%, respectively. These differences are smaller than the confidence interval of the baseline (± 6.3 points).

For this model, supplying the list of files to edit does not make the tasks easier. We believe the conclusions that can be drawn from this ablation are limited, however. While telling the agent where to edit allows the agent to focus on that part of the codebase earlier on in the trajectory, the agent still has to find and read the code that these files depend on to write a correct fix that is coherent with the codebase.

Reviewer_GWFmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SWE-Bench-XL, a benchmark for evaluating code agents on repository-level tasks in very large codebases. It contains 100 tasks from four large open-source repositories, with repository sizes substantially exceeding existing SWE benchmarks. Tasks are converted from merged pull requests into natural-language problems, containerized environments, and FAIL-TO-PASS / PASS-TO-PASS test oracles.

The authors evaluate five models under OpenHands and analyze performance differences across repositories and agent trajectories. The results show that agents perform more searches and visit more directories on SWE-Bench-XL, suggesting that the benchmark exercises codebase navigation and context management more strongly.

Critical strengths

It fills an important evaluation gap. SWE-Bench-XL is substantially larger than existing benchmarks and directly tests search, localization, and context management in large repositories. The tasks are grounded in real software engineering. They are mined from merged pull requests in real open-source projects and packaged in reproducible containerized environments. The analysis goes beyond final success rates. Search counts, visited directories, first-write positions, and other trajectory metrics provide useful behavioral information.

Critical weaknesses

The task and repository coverage is limited. The 100 tasks come from only four repositories, with a large fraction from VS Code and Elasticsearch. The results may depend strongly on specific repositories, languages, and build systems. Synthetic problem statements and test oracles introduce validity concerns. The benchmark relies on an agent-driven pipeline to generate descriptions and test oracles, but provides limited information about human review, error rates, test coverage, or possible leakage of the reference solution. Passing the oracle may not always imply that the original issue was correctly resolved. The link between repository size and necessary navigation is not fully established. More searches and directory visits show different agent behavior, but do not prove that broad navigation or cross-component understanding was necessary for solving the tasks. The evaluation uses a single agent harness. The results may conflate benchmark difficulty with OpenHands-specific behavior, and generalization across harnesses remains untested.

Questions for authors

How many of the 100 tasks were manually reviewed, and what are the error rates of the synthesized problem statements and test oracles? How do the authors ensure that the generated oracles do not leak or overfit the reference patch? Do the authors have task-level evidence that successful solutions actually require cross-directory, cross-component, or broad repository navigation? Why were tasks primarily collected from January to May 2026? How are contamination risks controlled relative to model training data, repository release dates, and evaluation dates? Can the benchmark be evaluated with additional agent harnesses or retrieval strategies to determine whether the findings are OpenHands-specific?

1 comment

We thank the reviewer for their time and constructive feedback.

W2/Q1/Q2. "How many of the 100 tasks were manually reviewed, and what are the error rates of the synthesized problem statements and test oracles? How do the authors ensure that the generated oracles do not leak or overfit the reference patch?"

Please see "Task construction and validation" in the general response.

W3/Q3. "Do the authors have task-level evidence that successful solutions actually require cross-directory, cross-component, or broad repository navigation?"

The proxy metrics in Figure 2 suggest that SWE-Bench-XL tasks require agents to understand a larger area of the codebase than tasks in existing benchmarks. For example, each edited file directly imports a median of 17 files in SWE-Bench-XL, against 8 in SWE-Bench-Verified, 8 in SWE-Bench-Pro and 4 in DeepSWE. In the median task, the direct imports of the edited files have a size of 184K tokens, against 34K, 47K and 30K.

The behavior of the agents points in the same direction (Table 4). With OpenHands and Claude Sonnet 5, an attempt on SWE-Bench-XL visits 10.3 directories and makes 22.9 searches on average, against 4.1 and 6.4 in SWE-Bench-Verified, 5.6 and 10.2 in SWE-Bench-Pro, and 6.1 and 16.1 in DeepSWE.

Without a ground-truth trajectory for each task, or an expensive manual analysis, we cannot be certain about which knowledge of the repository a successful solution requires, but our proxy metrics point in that direction.

W4/Q6. "Can the benchmark be evaluated with additional agent harnesses or retrieval strategies to determine whether the findings are OpenHands-specific?"

We have reproduced the experiment with an additional harness (OpenCode). The revised paper includes the results in Section 4.

Q4/Q5. "Why were tasks primarily collected from January to May 2026? How are contamination risks controlled relative to model training data, repository release dates, and evaluation dates?"

We collected pull requests from January to May 2026 because this was the most recent period available when we built the benchmark. The original PRs were merged between January 2 and May 11, 2026. We ran the evaluation in September 2026. While this period does not exclude contamination solely based on the knowledge cutoff dates of the models, there are limiting factors to it.

  • The problem statements and the test oracles are not available online as they were generated by our pipeline. A model can have seen the source pull request, but not the task and the oracle.
  • For Claude Sonnet 5, the model of our cross-benchmark comparison, the knowledge cutoff is January 2026. Its pass@1 is 23.7% on the 20 tasks whose original PR was merged before the cutoff and 22.2% on the other 80 tasks. In each of the four repositories, the tasks before the cutoff do not have a higher pass@1.

We also include canary strings in the paper and in all released data in an attempt to prevent future benchmark leakage.

Reviewer_SC53OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors present SWE-Bench-XL, a benchmark to measure the ability of coding agents to navigate large codebases with substantially more content. They demonstrate that current benchmarks target small codebases with potentially large reviews but insufficiently probe the ability to navigate a large codebase and its dependencies carefully. Across four repositories, no one model clearly dominates, and the tasks required from SWE-Bench-XL force more navigation on the part of tested models.

Critical strengths
  1. Well-motivated: the problem is clearly identified as targeted edits in large codebases, and is generally novel. The motivation connecting to context windows, relationship to industry codebases, and sizes of current public benchmarks is interesting, sound, and important. After reading the paper I not only agree that codebase navigation has not been sufficiently probed but feel surprised that other benchmarks don't pose a similar question. I like the use of source-code tokens to measure codebase size.
  2. Sound task pipeline: the identification of PRs to fix, gold-fixes for oracles, and composition of task statements appear to qualify with high fidelity the expectations for a standard coding benchmark. However, I have reservations about the tests used to grade task performance, and wonder if these tests might be miscalibrated since they are generated by the same models which apparently struggle to work in these codebases.
  3. Well-written: the writing is crystal clear, metrics are defined correctly and in the right places, and figures/tables are informative. I appreciated the introduction of an example change to a code-base.

My overall assessment is that this is a valuable benchmark measuring an under-represented but deeply relevant capability, but the analysis of results is comparatively lackluster, especially for more precise details. Some of these deficits (such as grading and choice of harness) warrant scrutiny of construct validity. More results data, especially for the ancillary metrics, would be helpful for understanding how performance on this benchmark actually looks and would resolve validity concerns. I recommend weak acceptance but am very willing to update upwards if primary weaknesses are addressed.

Critical weaknesses
  1. I want more results data. The paper defines nine different metrics but only shares the per-model pass@1 rate per-model. Table 4 only references a single model (or an aggregate? not clear, assumed Sonnet) for comparison across tasks. I did not catch any CIs reported for any result-based number. I'm left with a lot of questions; not ones necessarily against the validity of the benchmark, but mostly important findings I'm curious about. This would be good material for an Appendix, which appears to be missing from the paper. None of these requires additional GPU/inference compute:

    A. I would like to see a per-model table with a breakdown of all nine metrics, one which is averaged and then another four for each repository. Bonus points if we can get a similar one comparing to the reference benchmarks the authors have selected. Are there any behaviors (like more navigations, ops/gold file, FW@1) which are predictive of task performance? Do different tested models have different profiles for navigation? How about for the repositories?

    B. Can I get some CIs? The authors collect 5 attempts per instance but then aggregate to produce an estimate of pass@1. What is the per-attempt standard error average across tasks? What about per-task? This seems relevant to understanding how consistent behaviors or attempts are for resolution, and is a good practice for scientific inquiry in general.

    C. Cost! If the codebases are large, then it surely must take a lot of tokens and effort for each model to complete its tasks. How efficient is each model?

    D. Task difficulty: are there any heuristics the authors may propose for classifying the difficulty of a targeted task? Across failed test cases, are they determined more by FAIL_TO_PASS or PASS_TO_PASS? What does the partial pass-rate look like (meaning no. of FAIL_TO_PASS test cases satisfied, as opposed to requiring satisfaction of all of them)?

  2. Only one harness tested: it seems like only a single harness, OpenHands, is tested. This is problematic since many of the tested models are optimized for the harnesses created by their developers, as opposed to OpenHands. Indeed other big-codebase evals test this explicitly and find large differences. Especially if the use of operations and tools is dependent on the harness, one might imagine that performance is drastically different.

    A. This makes me wonder about per-repository rank volatility of the results, as well as the unintuitive ordering of model performance (Fable 5.1 would intuitively be at the top). Can you try a limited run where you compare the use of coding harnesses (one model developer-based, and another open one like Pi) and report findings?

    B. Compaction here seems especially important. I understand that the models do not read full codebases, and that the point of the task is the ability to do targeted interventions. Nonetheless, for a model to get this targeting from fresh context would likely require a lot of searches, as your navigation stats suggest. Context compaction is a unique confounder worth addressing given the task the authors try to solve, and it may be part of the reason that performance deflates. How frequent is compaction in attempts to solve tasks on this repository, and is it substantially larger than that of previous evals?

  3. Grading tasks: the description for how task scores are collected is unusual, given that the stated premise of the paper is that recent models struggle to work in large codebases. My understanding: an agent (which one?) is given the task description, the original codebase, and the desired behavior, and made to design FAIL_TO_PASS test cases (while PASS_TO_PASS is recycled from the current repository). The FAIL_TO_PASS cases are used if the reference solution passes them while the original solution fails them. This leaves a few questions:

    A. Basic information: how many FAIL_TO_PASS cases were created per task? How many PASS_TO_PASS? Does this vary across repositories or tasks? For multi-point desiderata (desired solution does X, Y, and Z without breaking A or B), how do the authors ensure that they have covered each point? Did the agent have access to the reference solution as it was designing test cases?

    B. Circular validation: how do the authors ensure that the FAIL_TO_PASS cases are truly separated from one another? If authors had an AI agent generate these test cases, then the long-horizon problem established as the claim in this paper would implicate the validity of the results. Are they categorically too easy? What AI was used to generate the tests?

    C. How do the authors know know that the original patch worked? People merge buggy PRs all the time, especially to open-source repositories. The patch might pass the test cases without actually fixing the problem because of some failure of imagination (see B).

  4. Weak related work section: the related works conduct a very short treatment of counterparts in the literature. It misses references to benchmarks which I found with a quick web search, such as SWE-CI. Also, some cited works like ProgramBench (Yang et al., 2026) and MirrorCode (Adamczewski et al., 2026) seem like relevant targets for cross-benchmark comparison. Both require rebuilding a repository -- a more demanding version of SWE-Bench-XL's task -- while LinuxFLBench (Zhou et al., 2026) targets performance on the Linux kernel, which is indeed massive. How do performance and the other ancillary measurements on SWE-Bench-XL compare to those tasks, especially LinuxBench?

Questions for authors

In addition to the questions asked above, the following came up during reading and writing the review.

  1. Token count metric: I appreciate the use of this, but I wonder if there could be more detail here. Could the authors share the distribution of tokens by file type in each repo? I wonder especially the ratios of documentation (code, docstrings, READMEs, etc.) to actual code, and how sizes are impacted by removing documentation. Also, how does the tokens-per-file distribution look? Lastly, how does this metric correlate with other conventional size measurements, such as filesize, lines of code, or number of files?

  2. What is the time limit? How impactful is the time limit on performance? Would increasing it improve performance on hard tasks?

  3. The authors say "[a]fter excluding error and missing outcomes" in line 315. What is meant by this? What are the errors and missing outcomes? How many were there? What happens if these get included in the results? Why were errors excluded if they were part of the task performance?

5 comments

We thank the reviewer for their time and constructive feedback.

Results A. "I would like to see a per-model table with a breakdown of all nine metrics, one which is averaged and then another four for each repository. […] Are there any behaviors (like more navigations, ops/gold file, FW@1) which are predictive of task performance?"

We added Appendix E. It provides the nine metrics for each of the ten model and harness pairs, overall and for each of the four repositories, with 95% confidence intervals. It also provides the metrics of Claude Sonnet 5 on all four benchmarks under both harnesses.

To find out if a behavior predicts success, we compared attempts on the same task. For each harness and benchmark with Claude Sonnet 5, we fit a linear probability model of success on one metric, with a fixed effect for each task and standard errors clustered by task. The fixed effects remove the differences in difficulty between tasks. We fit each metric separately because several metrics use the same counts (for example, Nav/W is calculated from reads, searches and writes). Each cell gives the change in the probability of success, in percentage points, for one standard deviation of the metric within a task, with its 95% confidence interval.

HarnessBenchmarkOpsReadsWritesSearchesDirsNav/WOps/GoldFW@AttemptsTasks
OpenHandsVerified+1.95 [-0.14, +4.04]+1.87 [-0.32, +4.05]+1.34 [-0.64, +3.32]+1.63 [-0.27, +3.53]+2.14 [+0.65, +3.64]-0.59 [-2.17, +0.98]+1.22 [-0.47, +2.91]+0.29 [-1.48, +2.06]2399494
OpenCodeVerified+1.65 [-0.25, +3.56]+2.01 [+0.03, +3.99]+0.56 [-1.30, +2.43]+1.44 [-0.23, +3.11]+0.30 [-1.11, +1.72]+0.80 [-0.49, +2.09]+1.16 [-0.56, +2.87]+0.44 [-1.10, +1.98]2471498
OpenHandsPro+0.27 [-1.46, +2.01]+0.26 [-1.53, +2.04]-0.09 [-1.78, +1.60]+0.01 [-1.57, +1.59]+1.39 [-0.23, +3.00]-0.36 [-1.98, +1.26]+0.39 [-1.06, +1.84]+1.93 [+0.25, +3.61]3020655
OpenCodePro-0.02 [-1.01, +0.98]+0.20 [-0.84, +1.24]-0.17 [-1.12, +0.79]-0.17 [-1.13, +0.78]-0.01 [-0.95, +0.92]-0.63 [-1.91, +0.65]+0.02 [-1.06, +1.11]+0.53 [-0.62, +1.68]3235650
OpenHandsDeepSWE-1.30 [-5.43, +2.83]-1.55 [-5.59, +2.49]-0.25 [-4.67, +4.17]-1.40 [-5.54, +2.73]-0.17 [-4.90, +4.55]-1.60 [-4.67, +1.47]-0.31 [-4.46, +3.85]+1.53 [-3.26, +6.33]415110
OpenCodeDeepSWE-0.90 [-5.41, +3.61]-0.41 [-4.66, +3.83]-1.56 [-6.16, +3.03]-0.26 [-4.19, +3.67]+0.60 [-3.31, +4.52]+0.95 [-2.82, +4.71]-0.77 [-4.61, +3.08]-3.00 [-5.84, -0.16]559113
OpenHandsXL-0.08 [-4.69, +4.53]-0.55 [-4.76, +3.66]-0.05 [-5.11, +5.02]+0.74 [-3.88, +5.36]+2.27 [-2.41, +6.95]-1.01 [-3.61, +1.58]+1.73 [-3.06, +6.53]+2.36 [-1.09, +5.82]46599
OpenCodeXL+0.75 [-3.57, +5.06]+1.43 [-2.84, +5.69]-2.13 [-6.93, +2.66]+0.83 [-2.88, +4.55]+1.48 [-1.57, +4.54]-2.48 [-5.78, +0.81]+0.60 [-5.06, +6.26]+2.09 [-0.23, +4.42]489100

Four of the 64 estimates are significant at the 5% level, with none remaining significant after the Benjamini-Hochberg correction for 64 comparisons. Thus, we do not find a behavior that predicts success within a task.

Results B. "Can I get some CIs?"

We have added confidence intervals to our results in the revised version of the paper.

Results C. "If the codebases are large, then it surely must take a lot of tokens and effort for each model to complete its tasks. How efficient is each model?"

Token usage varies across both models and harnesses. On SWE-Bench-XL, Sonnet 5 has the highest average total token usage under both harnesses, while GPT-5.6 Terra has the lowest. The per-model means are:

ModelOpenHands: total tokens (M)OpenCode: total tokens (M)
Claude Sonnet 53.1383.974
GPT-5.6 Sol1.7671.467
Claude Fable 5.12.1651.906
Claude Opus 51.7241.559
GPT-5.6 Terra1.4341.425

Holding the model fixed, Sonnet 5 uses more tokens per attempt on SWE-Bench-XL than on Verified or Pro: 3.138M versus 0.854M and 1.350M under OpenHands, and 3.974M versus 0.640M and 1.228M under OpenCode. However, usage is not uniformly higher than on other difficult benchmarks: Sonnet 5 averages 3.253M tokens on DeepSWE under OpenHands and 7.332M under OpenCode.

Results D. "Task difficulty: are there any heuristics the authors may propose for classifying the difficulty of a targeted task? Across failed test cases, are they determined more by FAIL_TO_PASS or PASS_TO_PASS? What does the partial pass-rate look like […]?"

We propose two simple, complementary dimensions for characterizing task difficulty: the scope of the required changes and the scale of the repository in which those changes must be located. DeepSWE and SWE-Bench-XL illustrate this distinction: DeepSWE asks for relatively large changes to smaller codebases, whereas SWE-Bench-XL asks for localized changes within very large codebases.

The observed agent behaviors are consistent with these different challenges. Holding Sonnet 5 and OpenHands fixed, DeepSWE elicits more writes per attempt than SWE-Bench-XL (15.2 versus 10.1), while SWE-Bench-XL elicits more searches (22.9 versus 16.1) and visits to more distinct directories (10.3 versus 6.1). All three directional patterns also hold under OpenCode. These observations suggest distinguishing implementation burden from localization and navigation burden, rather than characterizing difficulty using repository size or patch size alone.

Appendix F provides the partial test pass rates of the failed attempts on SWE-Bench-XL, for all ten model and harness pairs. Of the 3,099 failed attempts, 2,298 fail only FAIL_TO_PASS tests, 37 fail only PASS_TO_PASS tests and 764 fail tests of both kinds. Thus, failures are mostly determined by FAIL_TO_PASS tests. When an attempt fails FAIL_TO_PASS tests, it passes 14.5% of them on average. When it fails PASS_TO_PASS tests, it passes 17.6% of them. The rates vary by repository: 39.6% of FAIL_TO_PASS tests in transformers and 4.4% in elasticsearch. Most attempts that fail both kinds of tests are on elasticsearch (695 of 764). There, a common failure is a change by the agent that stops the code from building, so that every test fails.

Harness A/B. "Can you try a limited run where you compare the use of coding harnesses (one model developer-based, and another open one like Pi) and report findings? […] How frequent is compaction in attempts to solve tasks on this repository, and is it substantially larger than that of previous evals?"

We have reproduced our initial experiment with an extra harness (OpenCode). Please see the new version of Section 4.

Context compaction is more frequent in SWE-Bench-XL than in previous benchmarks in our Sonnet 5 runs with OpenHands.

BenchmarkAttempts with compactionTasks with compaction
SWE-Bench-Verified13.7%29.0%
SWE-Bench-Pro25.4%48.5%
DeepSWE59.6%96.5%
SWE-Bench-XL70.1%97.0%

The compaction rates also vary depending on the model.

ModelAttempts with compactionTasks with compactionCompaction events
Claude Sonnet 570.1%97.0%727
GPT-5.6 Sol66.7%91.0%690
GPT-5.6 Terra48.2%77.0%325
Claude Fable 5.135.5%62.0%294
Claude Opus 525.6%54.0%187

We see context compaction as a standard mechanism in coding agents that lets an agent continue to work when the trajectory does not fit in the context window of the model, and the harnesses that developers use have it enabled by default. Both harnesses in our evaluation use it (Appendix C). This is one extra aspect that SWE-Bench-XL, with long tasks in large codebases, exercises.

Grading A. "Basic information: how many FAIL_TO_PASS cases were created per task? How many PASS_TO_PASS? […] Did the agent have access to the reference solution as it was designing test cases?"

Each task has between 1 and 33 FAIL_TO_PASS tests (median 4) and between 1 and 12 PASS_TO_PASS tests (median 3). The table provides the number of tasks for each count.

Tests in the task12 to 34 to 67 to 1011 or more
FAIL_TO_PASS2029281112
PASS_TO_PASS194027113

The median number of FAIL_TO_PASS tests is 1 in vscode, 3 in elasticsearch, 5 in typescript and 6 in transformers.

To cover each requirement, the statement writer compares its test plan with the statement in both directions: each check must follow from the statement, each requirement should have a check. The test writer does not see the reference patch directly. It has access to a summary of the solution and, during validation, it sees how its tests ran on the patched code.

More information can be found in the new Appendix B.

Grading B. "If authors had an AI agent generate these test cases, then the long-horizon problem established as the claim in this paper would implicate the validity of the results. Are they categorically too easy? What AI was used to generate the tests?"

The tests were written by Claude Sonnet 4.6. One of the main motivations behind augmenting/generating test cases is the goal of constructing a strong oracle for each task, which is harder to do if we were to only rely on the tests available in the original repo. Writing them is also an easier problem than solving the task. The test writer is told which behavior is required and can iterate based on testing the ground-truth patch. The evaluated agents, however, only get the statement and have to find the code.

Grading C. "How do the authors know that the original patch worked? People merge buggy PRs all the time, especially to open-source repositories."

The tasks do not depend on the pull request having fixed its original issue. The statement and the FAIL_TO_PASS tests are written from the code change, so each task asks the agent to reproduce the behavior that the reference patch introduces. The reference patch is correct for the task by construction.

Related work. "It misses references to benchmarks which I found with a quick web search, such as SWE-CI. […] How do performance and the other ancillary measurements on SWE-Bench-XL compare to those tasks, especially LinuxBench?"

We have added SWE-CI to the related work (Section 6). SWE-CI measures how well an agent maintains a codebase across a sequence of commits. It focuses on long-horizon codebase maintenance and not necessarily on codebase size.

ProgramBench and MirrorCode evaluate agents on implementing programs from scratch, with the documentation and the original binary as the specification. They do not evaluate how agents operate in an existing codebase. Thus their results and our navigation metrics do not necessarily measure the same capabilities.

LinuxFLBench is the nearest to SWE-Bench-XL in codebase size, but its task is fault localization only. The agents find the faulty locations and do not write patches. This task is interesting, but it is different from the end-to-end tasks of SWE-Bench-XL.

Token count. "Could the authors share the distribution of tokens by file type in each repo? […] how sizes are impacted by removing documentation. Also, how does the tokens-per-file distribution look? Lastly, how does this metric correlate with other conventional size measurements, such as filesize, lines of code, or number of files?"

The sizes in the paper count only source-code tokens. They do not include documentation, READMEs, configuration files or data files.

We added Appendix A with a discriminated count of codebase sizes. It provides the number of tokens and the number of files for each file type, for each benchmark and for each SWE-Bench-XL repository. In SWE-Bench-XL, documentation is 5.2% of all tokens. In the process of expanding this analysis we also found a minor bug, which we report in the general response to all reviewers.

Figure 3b shows the distribution of tokens per file. Files in SWE-Bench-XL have a similar size to files in the other benchmarks (on average 1,655 tokens per file, against 1,376 to 1,899). This means that SWE-Bench-XL codebases are larger mainly because they contain more files (median task has 18 to 138 times more source files). Over all 1,444 tasks, the Spearman correlation of tokens is 0.995 with file size in bytes, 0.991 with lines of code, and 0.876 with the number of files. Table 6 provides all measures for each benchmark and each SWE-Bench-XL repository.

Reviewer_tY4yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents SWE-Bench-XL, a benchmark consisting of 100 reproducible software engineering tasks collected from four very large open-source repositories. The authors aim to evaluate the capabilities of coding agents in navigating and understanding large codebases, which are not sufficiently represented in existing benchmarks. The evaluation of five frontier models demonstrates that these tasks remain challenging, with pass@1 ranging from 22.5% to 47.6%. The authors also report that agents perform more codebase searches and visit more directories compared to existing benchmarks. SWE-Bench-XL is publicly released as reproducible Harbor tasks, making it accessible for future research and evaluation.

Critical strengths

This paper addresses an interesting and relevant limitation of existing software engineering benchmarks, particularly their limited coverage of very large repositories. Real world software engineering is done often with large mono repos or large combinations of many repos and assessing the capabilities of LLM agents on these repositories is very important. The authors have evaluated five frontier models and provided comparisons with existing benchmarks, offering useful insights into the behavior of coding agents when working with larger codebases. The release of SWE-Bench-XL as reproducible Harbor tasks is a valuable contribution, especially considering the increasing adoption of Harbor for coding-agent evaluation and RL training.

Critical weaknesses

My three main concerns with the paper are: I am not entirely convinced that repository size is a sufficient indicator of codebase complexity. Having more files and tokens does not necessarily mean that a task requires more complex reasoning or a deeper understanding of the repository. I would expect a more explicit justification of how repository size relates to the capabilities the benchmark aims to evaluate. The authors highlight that agents perform more codebase searches and visit more directories than in existing benchmarks. However, I do not see why these metrics necessarily indicate that the benchmark better evaluates codebase navigation. More searches could simply result from larger repositories or differences in the tasks themselves, without necessarily reflecting greater complexity. The relationship with existing benchmarks, particularly DeepSWE, could be better justified. The authors demonstrate that SWE-Bench-XL contains substantially larger repositories, but it is unclear why extending an existing benchmark such as DeepSWE would not be sufficient. I would also expect a clearer discussion of whether the proposed benchmark introduces fundamentally different challenges beyond repository size.

Questions for authors

I would like to ask two questions to the authors: First, how do the authors define and measure codebase complexity? Could you provide evidence that repositories containing more files and tokens actually require more complex reasoning rather than simply more navigation? Second, why are the number of searches and visited directories considered good indicators of benchmark quality? How can we distinguish navigation caused by repository size from navigation required by the actual task?

1 comment

We thank the reviewer for their time and constructive feedback.

W1/Q1. "I am not entirely convinced that repository size is a sufficient indicator of codebase complexity. […] Could you provide evidence that repositories containing more files and tokens actually require more complex reasoning rather than simply more navigation?"

We agree that repository size is not a sufficient indicator of codebase complexity per se. We believe, however, that repository size is an important dimension that software engineering benchmarks should capture and that the community benefits from benchmarks, like SWE-Bench-XL, that increase the coverage of software engineering environments not captured by existing benchmarks. Without sufficient coverage of the different dimensions of software engineering tasks it is harder to understand how coding agents behave and perform when deployed and to make informed decisions when developing them.

W2/Q2. "Why are the number of searches and visited directories considered good indicators of benchmark quality? How can we distinguish navigation caused by repository size from navigation required by the actual task?"

We agree that the number of searches and the number of directories alone do not indicate benchmark quality. We also do not separate navigation caused by the size of the repository from navigation required by the task as it is not clear how to do so since solving a task and navigating the codebase in which that given task is done are intertwined.

The goal of reporting those statistics is to better understand the differences in behavior in SWE-Bench-XL compared with the behavior of the same coding agents in existing benchmarks. Our intuition is that the differences in behavior show how SWE-Bench-XL exercises coding agents in different ways than existing benchmarks.

W3. "It is unclear why extending an existing benchmark such as DeepSWE would not be sufficient. I would also expect a clearer discussion of whether the proposed benchmark introduces fundamentally different challenges beyond repository size."

DeepSWE is composed of manually written tasks in small codebases (median 0.29M source-code tokens). This manual method does not easily scale to very large codebases. To write a new problem statement and verifier, an engineer must know the codebase well, which is much more difficult and expensive in a codebase of the size of the ones we include. For this reason, we included tasks based on real pull requests that the maintainer of the repositories already reviewed and merged.

The primary goal of SWE-Bench-XL is to understand how agents operate in very large codebases, and how their behavior differs from the behavior in the smaller codebases of existing benchmarks. Our analysis (Section 2.2) shows that the tasks we include are different. The median SWE-Bench-XL codebase is 121 times larger than the median DeepSWE codebase. The median SWE-Bench-XL patch changes 2 source files and 51 lines. The median DeepSWE patch changes 6 source files and 642 lines, and 25% of the source files in the patches are actually new files (4% in SWE-Bench-XL). DeepSWE asks for large changes to small codebases. SWE-Bench-XL asks for localized changes in very large codebases.

The trajectories also have a different shape (Table 4). With Claude Sonnet 5, the two benchmarks have a similar number of operations for each attempt (61.9 and 58.3). On SWE-Bench-XL, however, the agent does more searches (22.9 against 16.1), visits more directories (10.3 against 6.1) and does fewer writes (10.1 against 15.2). This means the agent uses more of the trajectory for navigation and less for implementation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (to all reviewers)

We thank all four reviewers for their time and helpful feedback. We answer the most common questions across all reviews in this general answer, and provide a changelog of the paper. We also summarize a bugfix we made during the rebuttal process. Additionally, we answer each reviewer's questions in more detail below their respective reviews.

Codebase size calculation fix. While expanding size calculation to additional file types (in reply to reviewer SC53) we found a minor bug which led to some build generated files to be included in the source-code token counts. We have now fixed this by adding an extra filter which only keeps files tracked by git at the base commit of each task. This bug affected all four benchmarks, and does not change our claims. The median codebase size of SWE-Bench-XL is 10x the median of SWE-Bench-Verified, which is the largest median of the compared benchmarks.

As a result, we have updated the numbers reported in the text throughout the paper, Figure 1, Table 1, and the methodology in Section 2.5.

Task construction and validation. Several reviewers asked how the tasks are built and checked. We added Appendix B which describes the pipeline we used. Separate agents build the environment, write the problem statement, and write the tests. The statement writer reads the reference patch and writes three things: a description of the behavior that must change, a summary of the solution, and a plan of what to test. Its instructions are to keep implementation details out of the statement and to plan only checks that the statement justifies. The test writer does not have the patch. It works from the statement, the summary and the plan, and we keep its tests only if they fail on the base commit and pass with the reference patch.

While we have spot checked our test creation methodology, we have not performed a human-led manual review of the SWE-Bench-XL instances because we are not experts in the codebase, would ourselves be biased reviewers, and are unable to hire human reviewers. We acknowledge that any benchmark instance may have false positives due to weak FAIL_TO_PASS tests as well as false negatives due to underspecified/implied implementations. Even benchmarks that have delivered substantial value to the research community (e.g. SWE-Bench-Pro, DeepSWE) were later found to have meaningful issues with testing which required updating the benchmark, restricting to a smaller portion (SWE-Bench to SWE-Bench-Verified), etc. We have described our test creation methodology extensively in Appendix B and believe our test creation approach is sound and is consistent with, or exceeds, established approaches. Further, if we union the OpenHands and OpenCode runs across all models, the union solvability across instances is 82%, indicating that SWE-Bench-XL instances are broadly possible. Further, none of the 82 solved instances are solved 100% of iterations indicating that our instances, while broadly solvable, are not trivially solvable.

Changelog.

  • Second harness: we ran all five models with OpenCode in addition to OpenHands. The abstract, the introduction, Tables 3 and 4 and Sections 3.1, 4.1 and 4.2 include the results (SC53, GWFm).
  • Confidence intervals: Tables 3 and 4 give 95% confidence intervals, and Section 3.2 describes how we calculate them. Section 4.2 now compares SWE-Bench-XL and DeepSWE with these intervals (SC53, AZ2L).
  • Navigation claims: the abstract, the contributions in Section 1 and Section 4.2 now describe the behavior of the agents without the claim that it measures navigation difficulty. Section 4.2 states that the comparison does not isolate the effect of codebase size and that the ratio metrics depend on the size of the patch and the number of writes. Section 2.2 states that direct imports are an upper bound on the context a solution needs (AZ2L, tY4y).
  • Codebase size: Section 2.5 states that we count only files tracked by git. The sizes in the text, Figure 1 and Table 1 are updated.
  • Related work: Section 6 cites SWE-CI (SC53).
  • Appendix A (new): codebase size by file type, tokens per file, and comparison with lines, bytes and files (SC53, tY4y).
  • Appendix B (new): the pipeline that builds the environments, problem statements and tests (SC53, GWFm, AZ2L).
  • Appendix C (new): settings of the two harnesses, including context compaction and the time limit (SC53, AZ2L).
  • Appendix D (new): time limits, errors and missing outcomes for each harness and model, and pass@1 with errors counted as failures (SC53, AZ2L).
  • Appendix E (new): the nine trajectory metrics for each model and harness pair, overall and for each SWE-Bench-XL repository, and for Claude Sonnet 5 on all four benchmarks (SC53).
  • Appendix F (new): partial test pass rates of the failed SWE-Bench-XL attempts, for FAIL_TO_PASS and PASS_TO_PASS tests, for each repository and each task (SC53).
  • Appendix G (new): input, output and total tokens per attempt for each benchmark, harness and model (SC53).
  • Appendix H (new): frequency of context compaction for each benchmark and each model (SC53).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.