acceptodds
Withdrawn from ICLR 2027

SWE-Climb: Benchmarking Coding Agents on Real Work in Million-Line Brownfield Codebases

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

15 views

Abstract

Real-world software engineering rarely starts from an empty repository. Most of it happens inside large, old, heavily modified codebases—brownfield systems accumulated over years of patches, where documentation is stale, configuration is scattered across layers, and behaviors that no document promises are silently depended upon. We present SWE-Climb, a benchmark that operationalizes the three kinds of deliverables engineers actually produce on such systems: explaining them (writing a developer tutorial for a concrete change), running them (authoring executable configs and scripts that drive the repository's real toolchain), and changing them (implementing a reasonable-size feature or fix while preserving old behavior). The current construction snapshot comprises 240 fundamental tasks (120 Tutorial, 120 Config/Script) over 131 real repositories (median 1.6M lines of code and 13.6 years of history, 20 languages), each derived from real upstream development evidence, with de-identified prompts and fully verifiable grading: per-fact rubrics grounded to source paths, offline containerized functional checkers, and fail-to-pass/pass-to-pass behavioral tests. We report three observations: (1) frontier agents leave large headroom—no agent resolves more than 41% of Config tasks, and 29% of them are resolved by no agent in any trial—and two independent grading channels, an LLM-judged rubric and a deterministic binary checker, disagree exactly where explaining and operating a system come apart: the best explainer resolves the fewest Config tasks; (2) running an evolved system is substantially harder than explaining it: every agent's Config resolved rate is 0.36–0.65 of its Tutorial score; (3) failures concentrate on load-bearing facts with no surface signal: for every agent, recall of legacy-invariant facts trails recall of fix-path facts, deep call chains and implicit compatibility paths are the hardest Config mechanisms, repository size barely predicts difficulty, and more tool calls do not buy more resolved tasks. A paired knowledge-injection experiment tests whether supplying exactly these facts closes the end-to-end gap. Alongside the tasks we release the graders, frozen images, version manifests, and full agent trajectories, together with the validity protocol (three-state grader closure, a no-repository probe, adversarial audits, and error-source accounting) that we argue benchmarks of naturally dirty systems require.

Primary area:
datasets and benchmarks
Keywords:
coding agents, software engineering benchmarks, brownfield codebases, agent evaluation, program understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FdsqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SWE-Climb evaluates coding agents on three types of maintenance tasks in large, mature repositories: explaining the system, operating its toolchain, and modifying code while preserving existing behavior. It combines fact-based LLM grading with executable and regression tests. Experiments report differences in agent rankings across task types, difficulties recovering legacy invariants, and substantial performance drops without repository access.

Critical strengths
  1. Practically relevant tasks. The focus on large, mature repositories and preservation of existing behavior reflects meaningful maintenance challenges. The curated tasks could provide a useful resource for evaluating repository-level agents.

  2. Attention to grading quality. Requirement grounded checks, alternative valid solutions, and separation of functional failures from infrastructure errors are sensible safeguards that could improve evaluation reliability.

Critical weaknesses
  1. Limited novelty and unclear evidence for the “climb.” The three rungs resemble established benchmark settings: repository understanding, executable tool use, and SWE-bench-style code modification. Since they use different tasks and metrics rather than matched stages of the same maintenance problem, it is unclear how they establish a progression from exploration to operation to evolution. I would appreciate clarification of what additional insight the combined framework provides beyond evaluating these three task families separately. At present, the contribution appears primarily to be their aggregation.

  2. Empirical validation could be strengthened. I had difficulty reconciling the OpenReview abstract with the PDF, as several experiments mentioned in the abstract do not appear in the manuscript. Exploration also uses GPT-5.5 both as the primary judge and as an evaluated model, while the alternative judge belongs to the same family; validation with independent judges would help address potential bias. Similarly, a shared harness controls the interface, but may interact differently with models trained for their native agent environments. The results therefore support conclusions about the tested model–harness combinations more directly than about the models themselves. Evaluation with current frontier systems, such as Astra-6 or Opus 5.5, would also help establish the benchmark’s present relevance. I appreciate the substantial evaluation cost, but even a representative subset tested with these systems in their native harnesses would be informative.

In my view, a compelling new code benchmark should demonstrate useful training or evaluation infrastructure, identify an important capability gap overlooked by existing benchmarks, or establish a harder test that exposes current frontier limitations. I would welcome clearer evidence of which of these contributions SWE-Climb offers; the current experiments leave this question unresolved.

Questions for authors
  1. What capability or insight does SWE-Climb capture beyond existing benchmarks? Are there matched tasks across the three rungs that substantiate the proposed “climb”?

  2. Could you reconcile the OpenReview abstract with the PDF, particularly the missing knowledge-injection experiment, and clarify which results belong to the submitted version?

  3. Is there human or cross-family judge validation showing that Exploration scores and rankings are robust to the choice of evaluator?

  4. To strengthen the empirical evidence, could you provide a representative subset evaluated with current frontier models in their native agent harnesses, or existing evidence that the main findings persist across harnesses?

Reviewer_sH2eOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

SWE-Climb evaluates coding agents on three aspects of software engineering – Exploration, Operation, and Evolution. It contains 120 exploration tasks, 120 operation tasks, and 57 evolution tasks from 130 Github repositories.

Critical strengths

The three way separation is helpful to pinpoint software engineering capabilities of the coding agents

Critical weaknesses
  1. The main concern over this paper is the writing. There is severe oversight by the authors and the paper seems to be entirely AI generated with several dense paragraphs and very hard to follow structure. There is an entire repeated paragraph (para 2 and 3) in the Introduction, while the key takeaways from the Introduction require one to read the full paper to start making sense. There are several terms (e.g., fundamental-track on Line 198, fix path, legacy invariant, F2P, P2P, etc.) that are introduced without ever defining them.
  2. The contribution seems incremental for every dimension. Evolution evaluates repository-level changes, like feature implementation and bug fixes, which overlap with SWE-bench [1] and its variants. Operation evaluates the ability to configure and execute an existing repository, similar to work done in BuildBench [2]. Exploration also follows closely to RepoProbe [3] which evaluates repository understanding using weighted technical checklists.
  3. The LLM-judge requires validation. GPT-5.5 serves as both an evaluated model and the main judge. Section 5.4 reports repeated scoring and a judge swap to GPT-5.4, which provide useful evidence of consistency. However, neither establishes agreement with human assessment or measures potential bias.
  4. Table 1 averages three runs per task but does not report error bars. Appendix I reports median differences of 5 to 8 points between runs on the same tasks. This makes estimates uncertain when interpreting close ranking models.

[1] Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Zhang, Zehua, et al. "Buildbench: Benchmarking llm agents on compiling real-world open-source software." arXiv preprint arXiv:2509.25248 (2025).

[3] Yang, Yuexi, et al. "RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists." Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering. 2026.

Questions for authors

See weaknesses above

Reviewer_RxtuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SWE-Climb, a benchmark designed to evaluate the maintenance capabilities of coding agents within large, long-evolving codebases. The benchmark categorizes tasks into three components: Exploration, which requires articulating the system knowledge necessary to implement a change; Operation, which involves producing verifiable results using existing toolchains; and Evolution, which entails implementing new features or fixing issues while preserving existing functionality. The foundational component comprises 120 Exploration tasks and 120 Operation tasks drawn from 130 curated upstream repositories; Exploration tasks are evaluated using a weighted score based on 2, 404 atomic facts. The Evolution component consists of 57 tasks from 54 repositories, requiring that both new behavioral tests and regression tests for existing functionality pass. The median codebase size is 1.63 million LoC for the foundational repository set and 1.34 million LoC for the Evolution repository set. The paper’s contributions include diverse interfaces for maintenance tasks, a knowledge-scoring mechanism grounded in source-code evidence, and an evaluation protocol that emphasizes requirements traceability, he identification of valid alternative solutions, and the attribution of failure causes. Experiments were conducted by evaluating six models on the foundational tasks, four of which were also tested on the Evolution tasks, with each task executed three times. Key findings indicate that model rankings vary by task type; Exploration scores drop significantly when repository access is restricted; and some agents, despite identifying relevant files or completing most of a new feature, still fail to account for conditional dependencies or inadvertently break existing functionality. The paper also provides data for re-evaluating models, swapping scoring models, and analyzing specific execution trajectories.

Critical strengths
  1. This paper transforms the maintenance of knowledge into verifiable objects, offering clear diagnostic value. A significant design choice in this work is the mapping of interpretation, execution, and modification to distinct deliverables, grounding the knowledge requirements of the Exploration phase in atomic facts substantiated by source code evidence. This approach allows for the decomposition of a failed final task into specific, actionable issues, such as identifying the correct entry point, determining configuration priorities, accounting for existing behavioral constraints, or establishing validation paths. The concept of equivalent evidence is also commendable; by allowing a single fact to be supported by various sources, such as source code, tests, or configurations, the framework accommodates diverse, yet valid, reasoning paths. As demonstrated by the Cypress and Neovim case studies, this fine-grained evaluation can pinpoint the gap between knowledge of isolated functions and a comprehensive understanding of overall system behavior. This is a key contribution, as it provides diagnostic insights that extend well beyond simple success-rate metrics.
  2. The paper’s scoring protocol rigorously addresses the challenges of ensuring evaluation validity within large, real-world code repositories. The framework incorporates several critical checks into the task-admission process. For the Operation phase, it mandates that a proposed solution must pass the reference test suite, fail when provided with incorrect inputs, and yield no result for empty inputs; it also ensures that functionally equivalent solutions, even those with different structural implementations, are accepted. For the Evolution phase, the protocol requires that the baseline and the reference patch produce a distinct shift in test outcomes, while also verifying that the (peer-to-peer/patch-to-patch) tests are sufficiently sensitive to mutations in existing behavior. Furthermore, for high-weight facts in the Exploration pphase, theprotocol requires an independent reviewer, unfamiliar with the upstream implementation, to confirm that the proposed solution is reproducible. These design elements are directly linked to the reliability of the scores as indicators of an agent's capabilities. The validation of equivalent solutions mitigates the risk of the scorer conflating a preference for a specific reference implementation with objective correctness; the classification of execution states helps distinguish between functional failures and infrastructure-related issues; and version binding facilitates the tracking of results following task modifications. Although the paper has yet to provide a comprehensive audit of these safeguards across the entire dataset, the protocol itself holds substantial value.
Critical weaknesses

W1. There is a discrepancy between the reported success rates and the stated statistical protocol; furthermore, the actual evaluation configuration is insufficient to reconstruct or verify the results.

Section 4.1 and Table 1 of the paper state that each model was run three times across 120 Operation tasks, with binary success rates reported. Following this protocol, the success rate should be the number of successful attempts (an integer) divided by 360. However, the 40.4% success rate reported for Opus 4.8 cannot be derived from this denominator, even when rounded to one decimal place. Specifically, 145 successes would yield approximately 40.3%, while 146 successes would yield approximately 40.6%. While this might simply be a typographical error in the table, it could also indicate undisclosed exclusions, re-runs, or weighting schemes; therefore, the authors should provide the actual numerators and denominators used.

This issue is directly related to how failure states are handled. Appendix A.3 categorizes missing answers, unexecuted answers, timeouts, environment errors, and functional failures, but it does not specify how these states factor into the final denominator used in Table 1, nor does it report the frequency of these states for each model. Calculating a functional success rate based only on successfully executed samples carries a different meaning than calculating a completion rate based on all task attempts.

Furthermore, although the paper repeatedly mentions that the budget and execution framework remained consistent, it fails to specify the actual limits, such as time, token counts, or tool-call budgets, nor does it provide the specific version of the framework or the complete prompts used during execution. While Appendix A.2 lists the fields that should be recorded in the manifest, these definitions are not a substitute for the full experimental configuration. For a benchmark whose primary contribution is the measurement of reliability, these omissions significantly hinder the interpretation and reproducibility of the results.

W2. While the stability of the scoring in the Exploration phase has been partially verified, there is insufficient evidence to confirm that these scores accurately measure the system's underlying knowledge.

In Section 5.4, the authors assess stability through repeated scoring and by substituting the scoring model which comparing results from a GPT-5.5 scorer versus a GPT-5.4 scorer. I believe this validation provides reasonable support for the experimental results. However, the consistency between two scoring models does not, in itself, prove that they are correctly evaluating the accuracy of source-based facts, the validity of paraphrased representations, or the severity of omitted information. The human review described in Appendix C focuses on the recoverability of ground-truth facts; it does not constitute an independent human evaluation of the agents' actual responses. The paper does not report human calibration results at the level of the responses themselves.

Table 4 shows that replacing the scoring model leads to variations in both the direction and magnitude of the scores. The average scores for GPT-5.5 and Doubao both increased by 2.2 points, whereas the score for DeepSeek dropped from 58.0 to 51.9, and those for MiniMax and Qwen decreased by 3.5 and 3.7 points, respectively. While this does not in itself prove the existence of bias, it suggests that a high overall correlation can still mask scoring discrepancies linked to response style or the specific model being evaluated.

Furthermore, the paper treats Exploration as an observable proxy for system knowledge, using it to discuss differences in model capabilities and fact types. The current evidence primarily indicates that models achieve different weighted rubric scores; it is not yet sufficient to confirm that these differences consistently reflect the completeness of the models' knowledge. Independent calibration, covering the factual accuracy of the agents' responses, the acceptability of alternative phrasings, and the penalties applied for hallucinations, is required. Additionally, it is necessary to verify whether the paper’s primary conclusions are sensitive to the specific weights and scoring models employed.

W3. The analysis of the three task categories and execution trajectories supports the existence of performance variations across different dimensions; however, it fails to clearly identify the knowledge bottlenecks or the specific stages of failure emphasized by the paper.

Section 4.1 explicitly acknowledges that Exploration and Operation utilize different sets of tasks. Consequently, while shifts in model rankings indicate that these two sets of tasks measure different performance aspects, they do not directly demonstrate a capability gap, a transition from knowing to operating, within a single maintenance task. The claim on page 2, that a unified hierarchical evaluation would misrank maintenance capabilities, lacks a reference to an independent, overarching standard for maintenance proficiency. Similarly, the lower success rate in the Evolution category cannot be attributed solely to the difficulty of knowledge acquisition, as the task content, output requirements, and the criteria for achieving a pass across all tests also differ.

The statistics presented in Section 5.3, concerning failures that occur even after the correct location has been identified, also require more precise measurement. In Appendices I.1 and I.2, file access is tracked exclusively through successful, dedicated Read calls. A hit on a file does not guarantee that the model has retrieved the specific code snippet corresponding to a given fact; conversely, source code information obtained via other tools is not fully captured by this metric. Consequently, the fact that 41.93% of the relevant items in Table 9 did not receive a perfect score indicates that file access is not equivalent to providing a complete answer. However, one cannot simply attribute all these instances to a failure in understanding following a successful localization. While the specific traces provided in the appendix offer insight into individual cases, they cannot substitute for an aggregate statistical breakdown of the stages at which failures occur.

Similarly, in Figure 4, the gap between legacy behavior invariants and modification paths ranges from 2.5 to 6.1 percentage points; however, the study does not report the uncertainty of these differences, nor does it account for the varying difficulty levels associated with different types of facts. While the examples from NetBeans and Flowable demonstrate that regressions in legacy behavior can lead to the rejection of a patch, the study does not quantify how frequently this mechanism contributes to the overall failure rate in the Evolution category. The absence of a comprehensive breakdown, distinguishing between failures in Fix-to-Pass tasks, Pass-to-Pass tasks, and both, makes it difficult to quantify the prevalence and actual impact of blind spots regarding legacy behavior.

W4. Task selection and prompt pruning based on model performance limit the scope of conclusions regarding repository dependency.

Appendix E notes that if at least two models successfully complete an Operation task without access to the repository, the task is modified to require more repository-specific information. Appendix D further states that the selection criteria for the Evolution category aim to ensure that the performance of the strongest model falls somewhere between the minimum and maximum observed values. While these choices help construct a benchmark with high discriminative power, they also mean that the final distribution of tasks is, a function of model performance. The paper does not explicitly clarify whether the models, computational budgets, or experimental runs used for this filtering process were independent of the final evaluation, nor does it report how the task distribution shifted as a result of this filtering.

Therefore, while the performance decline observed in the no-repository condition confirms that the filtered task set is indeed dependent on repository information, the magnitude of this decline cannot be directly extrapolated to represent the general degree to which real-world maintenance tasks rely on source code access. Furthermore, a drop in performance upon the removal of source code does not rule out the possibility that models are relying on memorized knowledge of public projects; source code access simultaneously alters both the cues available for retrieval and the conditions under which an answer can be verified. In fact, Appendix E acknowledges that eight tasks remain solvable by at least two models even in the absence of repository access. Furthermore, the MongoDB examples remove terminology related to mechanisms such as rollback and oplog, rather than merely stripping away final patch identifiers. The paper needs to explain why such information constitutes a leaked clue that ought to be removed, and whether the remaining context is still sufficient to address the actual requirements of the task. Appendix D mentions stratifying data by training cutoff dates and implementing controls, specifically restricted context and greenfield-hollow settings, yet it fails to provide the corresponding results. Consequently, it is difficult to disentangle the respective impacts of inherent maintenance difficulty, the reduction of prompt information, and the selection of tasks in the current experiments.

Questions for authors
  1. Please provide the actual number of successful attempts, the total number of attempts, and the counts for each type of failure/abnormal state for every model listed in Table 1. How was the 40.4% success rate for Opus calculated, given 120 tasks with three runs each? Are missing answers, unexecuted answers, and timeouts included in the final denominator? What are the rules for re-running tasks?

  2. Were the generated Exploration responses evaluated blindly by independent human reviewers? If so, please report the inter-rater reliability (fact-by-fact scoring), the false-positive rate regarding valid alternative phrasings, and the consistency in penalizing hallucinations. If different weights or an alternative evaluation model were used, would the relative performance between the primary model and the legacy behavior invariant baseline remain consistent?

  3. Regarding inferences across task types: Are there instances where the same repository and behavioral requirement are associated with Exploration, Operation, and Evolution tasks, allowing for an analysis of the relationship between knowledge scores and subsequent execution success? Can the existing data distinguish between incomplete knowledge and failures in implementation, tool usage, or verification?

  4. Regarding failures occurring after localization: For the file hits mentioned in Table 9, is it confirmed that the relevant lines of code were actually included in the model's context window? Does this account for source code read via search tools or terminal commands? What are the specific values and uncertainty ranges for the two correlation coefficients mentioned in Appendix I.1?

  5. Regarding the preservation of legacy behavior: Among all Evolution attempts, what proportion passed the functional-to-functional tests but failed the property-to-property tests? What are the proportions for cases where both test types failed, versus cases where only the F2P tests failed? These statistics are critical for determining whether the preservation of legacy behavior acts as a primary bottleneck.

  6. Regarding task filtering and repository-free (no-repo) controls: Which models and runs were used for task filtering, and how were they isolated from the final evaluation? For the repository-free experiments, how many runs were retained versus excluded for each model? Which specific models, tasks, and versions are covered by the 2, 400 trajectories in Appendix E, and what were the results of the other control experiments mentioned in Appendix D?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.