acceptodds
Under review as a conference paper at ICLR 2027

SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle

Abstract

As autonomous code agents move toward end-to-end software development, evaluating their practical autonomy becomes critical. Current benchmarks hide friction by testing agents in pre-configured environments, and their static evaluation pipelines frequently fail when parsing fully autonomous trajectories. SWE-Cycle addresses these limitations with 489 rigorously filtered instances. It evaluates agents across three isolated tasks, including environment reconstruction, code implementation, and verification test generation, as well as an end-to-end FullCycle task that integrates all three. In FullCycle, one agent starts from a bare repository and carries its own environment, code, tests, and session state across the issue-resolution process. To reliably assess these complex execution paths, we developed SWE-Judge. By combining static code review with dynamic testing, this execution-capable evaluator reliably assesses functional correctness and corrects systematic misjudgments introduced by rigid script-based evaluation pipelines. We evaluate code agents powered by seven state-of-the-art LLMs across these four tasks. The results reveal a sharp drop in solve rates when transitioning from isolated tasks to FullCycle execution, exposing critical bottlenecks in handling cross-phase dependencies and maintaining code quality. Together, SWE-Cycle and SWE-Judge provide a comprehensive framework for accurately measuring the end-to-end capabilities of autonomous software agents.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Benchmark, Code Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bCTjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SWE-Cycle, a benchmark of 489 issues curated from SWE-bench Verified, Pro, and Multilingual. It evaluates environment reconstruction, code implementation, and verification test generation both independently and within a single FullCycle agent session. The paper also proposes SWE-Judge, an LLM-driven evaluator agent that combines static artifact inspection with dynamic execution and reference-assisted test refinement. Experiments with seven solver models examine task performance, differences between isolated and integrated execution, and cross-phase dependencies. Validation includes human auditing, a comparison of two evaluator backbones, and phase-removal ablations. The reported results highlight a gap between high average scores and perfect completion across all lifecycle dimensions.

Critical strengths

S1. A practically relevant evaluation scope with a useful paired design.

Requiring an agent to produce its own environment, implementation, and verification tests addresses aspects of repository maintenance that patch-only scoring does not explicitly assess. Evaluating the same issues in isolated and FullCycle settings provides a structured basis for examining how these capabilities interact, making the benchmark potentially useful for diagnosing failures beyond final patch correctness.

S2. An execution-grounded evaluator supported by substantial initial validation.

SWE-Judge supplements artifact inspection with actual execution and reference-assisted verification rather than relying exclusively on textual judgments. The 946 human inspections, two-backbone evaluator comparison, and documented failure cases provide meaningful initial evidence for its usefulness in identifying limitations of rigid script-based evaluation, although they do not fully resolve the reliability concerns below.

Critical weaknesses

W1. The reliability of the LLM-driven agentic evaluator and repeated solver execution is not sufficiently established.

SWE-Judge is an LLM-driven evaluator agent that autonomously inspects artifacts, invokes tools, executes tests, and may generate additional verification scripts. Its judgments can therefore depend on both the backbone model and the agent’s evidence-gathering and execution trajectory. Appendix D compares Claude-Opus-4.5 and GPT-5.4 on 414 evaluation trials, but does not establish the stability of repeated evaluations of identical submission snapshots by the same evaluator agent. Sensitivity to evaluator capability tiers and model-version changes is also insufficiently characterized, and similar aggregate solve rates can conceal substantial instance-level disagreement.

Separately, although Appendix F reports 95% confidence intervals, the manuscript does not clearly document repeated independent solver runs or separate solver variability from evaluator variability. Neither pass@k nor pass^k is reported for k>1k>1. These omissions limit the evidence for repeatable measurement and consistently successful autonomous execution, both of which are central to the paper’s claims.

W2. The incremental measurement contribution and robustness to data contamination require stronger evidence.

DeepSWE already evaluates autonomous repository-level engineering and analyzes agent self-verification. Explicit environment reconstruction is a potentially meaningful extension, but its independent effect on task completion remains unclear. The −Env ablation in Appendix J reports changes in remaining-phase scores rather than success rates under a common acceptance criterion, while also acknowledging changes in context, available artifacts, and effective budget. It therefore does not establish whether environment reconstruction introduces a substantial, statistically supported obstacle to completion or which additional failures it reveals beyond preconfigured evaluations.

Furthermore, all issues are inherited from existing benchmarks, while contamination screening relies on DeepSeek-V3.2 attempting tasks without their issue descriptions. Failure under this restricted setting does not establish that a model will not recall a solution once the issue is provided, nor that the screening result generalizes across model families. The uncharacterized residual contamination further limits confidence that the benchmark measures previously unseen problem-solving rather than prior exposure.

Questions for authors
  1. How stable are instance-level scores and Solve decisions when identical submission snapshots are repeatedly assessed by the same LLM-driven evaluator agent and by evaluator agents using backbones from different capability tiers and model versions?

  2. Can you provide pass@k and pass^k for k>1k>1, with confidence intervals from repeated independent solver runs and evaluator variability measured separately?

  3. What evidence supports low residual contamination across the evaluated model families beyond the single-model, no-issue-description screening procedure?

  4. Under matched tasks, budgets, and code-and-test acceptance criteria, what statistically supported change in end-to-end success is attributable specifically to requiring environment reconstruction rather than providing a preconfigured environment?

3 comments

Dear Reviewer bCTj,

We thank you for recognizing the value of our paired task design and execution-grounded evaluation. Your comments help us clarify measurement repeatability, the incremental contribution of SWE-Cycle, and the evidence concerning data contamination. We address each concern below.


Weakness 1: The reliability of the LLM-driven agentic evaluator and repeated solver execution is not sufficiently established.

SWE-Judge is an LLM-driven evaluator agent that autonomously inspects artifacts, invokes tools, executes tests, and may generate additional verification scripts. Its judgments can therefore depend on both the backbone model and the agent’s evidence-gathering and execution trajectory. Appendix D compares Claude-Opus-4.5 and GPT-5.4 on 414 evaluation trials, but does not establish the stability of repeated evaluations of identical submission snapshots by the same evaluator agent. Sensitivity to evaluator capability tiers and model-version changes is also insufficiently characterized, and similar aggregate solve rates can conceal substantial instance-level disagreement.

Separately, although Appendix F reports 95% confidence intervals, the manuscript does not clearly document repeated independent solver runs or separate solver variability from evaluator variability. Neither pass@k nor pass^k is reported for (k>1). These omissions limit the evidence for repeatable measurement and consistently successful autonomous execution, both of which are central to the paper’s claims.

Response 1: Thank you for identifying these important questions concerning evaluation repeatability and variance across runs of the solve agent. We summarize our available evidence and concrete plans for expanded variance measurement below.

First, SWE-Judge operates under explicit, dimension-specific 0–2 scoring rubrics (Appendix B) that specify exact Static and Dynamic assessment targets, score thresholds, and deterministic rules for handling conditions such as environment failures and vacuous tests, minimizing discretionary interpretation. In an initial repeatability study involving 100 submissions evaluated independently twice each by the same judge agent, we observed 92% dimension-level exact agreement and 97% Solve agreement between evaluations, providing direct initial evidence of instance-level repeatability under our rubric.

Second, we conducted an independent blind human evaluation of 300 submissions covering all four task types (Env, Impl, TestGen, and FullCycle), where two annotators scored submissions independently without seeing SWE-Judge outputs and a third annotator adjudicated disagreements against the raw artifacts. SWE-Judge achieved 94.3% dimension-level exact agreement and 98.3% Solve agreement with the adjudicated human labels, providing complementary instance-level evidence of judgment accuracy.

Finally, the judge agent executes inside the solve agent’s container to inherit its live environment and artifacts (Section 4.1). To explicitly separate the variability of the solve agent from that of the judge agent across multi-run trials, we are archiving pre-evaluation container snapshots to launch repeated evaluations from independent clones on fixed outputs from the solve agent, while conducting repeated independent runs of the solve agent to report pass@3 and pass^3 in the revised manuscript.

Weakness 2: The incremental measurement contribution and robustness to data contamination require stronger evidence.

DeepSWE already evaluates autonomous repository-level engineering and analyzes agent self-verification. Explicit environment reconstruction is a potentially meaningful extension, but its independent effect on task completion remains unclear. The −Env ablation in Appendix J reports changes in remaining-phase scores rather than success rates under a common acceptance criterion, while also acknowledging changes in context, available artifacts, and effective budget. It therefore does not establish whether environment reconstruction introduces a substantial, statistically supported obstacle to completion or which additional failures it reveals beyond preconfigured evaluations.

Furthermore, all issues are inherited from existing benchmarks, while contamination screening relies on DeepSeek-V3.2 attempting tasks without their issue descriptions. Failure under this restricted setting does not establish that a model will not recall a solution once the issue is provided, nor that the screening result generalizes across model families. The uncharacterized residual contamination further limits confidence that the benchmark measures previously unseen problem-solving rather than prior exposure.

Response 2: Thank you for asking us to clarify our incremental measurement contribution and contamination controls. SWE-Cycle’s contribution extends beyond adding lifecycle stages: it constructs paired Isolated and FullCycle settings for the exact same 489 issues, compares phase-level performance across settings, and diagnoses cross-phase dependencies through phase-removal ablations (Section 4.5, Appendix J). Together with the dataset, SWE-Cycle provides a reusable evaluation methodology that can extend to broader benchmarks to systematically diagnose how isolated capabilities translate into full-lifecycle autonomy.

Effect of environment reconstruction. The table below compares end-to-end success (Solve) rates across paired runs in Appendix J between FullCycle (requiring full credit across all six Env, Code, and Test Static/Dynamic sub-dimensions) and −Env (requiring full credit across the four remaining Code and Test sub-dimensions):

Solve AgentFullCycle success rate−Env success rate
Claude-Sonnet-4.612.35%20.37%
GLM-5.113.55%19.30%
Pooled12.95%19.84%

Success rates are consistently higher under −Env, showing that satisfying full-lifecycle requirements—including environment reconstruction—poses a substantial additional obstacle to autonomous completion. Inspecting paired evaluation trajectories reveals concrete environment-delivery failures that preconfigured evaluations miss:

  • Incomplete dependency setup (OpenLibrary, Claude-Sonnet-4.6, instance containing ba3abfb6): The FullCycle submission’s setup script relied on a nonexistent Miniconda installation, while the system Python environment lacked required packages including requests and pytest. The judge agent had to install missing dependencies and remove conflicting pytest plugins before verification could proceed.
  • Unusable designated runtime (Matplotlib, Claude-Sonnet-4.6, matplotlib__matplotlib-23412): Starting Python in the designated testbed environment returned Invalid argument, whereas the required packages were installed in the base environment. The judge agent had to fall back to base to import the project and collect tests; thus, although functional tests could run in a fallback environment, the submission failed to deliver the required working runtime.

These cases demonstrate that SWE-Cycle additionally assesses whether a solve agent can deliver a usable designated runtime, install required dependencies, and leave behind a verifiable execution state—capabilities omitted when evaluating code and tests solely inside preconfigured containers, yet essential for autonomous completion without external intervention.

Contamination screening. Because pretraining corpora across evaluated model families are undisclosed, we mitigate memorization risks at both the task-filtering and task-formulation levels. First, our no-issue-description screening with DeepSeek-V3.2 removes 128 instances exhibiting direct memorization signals (Section 3.1, Appendix A). Second, FullCycle requires the solve agent to autonomously configure a bare environment, implement the fix, and construct discriminative verification tests under joint static–dynamic grading—tasks that cannot be passed merely by recalling an isolated code patch. Moreover, because SWE-Cycle provides an extensible task-construction methodology alongside the dataset, its pipeline can be directly applied to private repositories or newly collected pull requests to continuously refresh evaluation sets.

Question 1: How stable are instance-level scores and Solve decisions when identical submission snapshots are repeatedly assessed by the same LLM-driven evaluator agent and by evaluator agents using backbones from different capability tiers and model versions?

Answer 1: As discussed in Response 1, our initial study of 100 submissions evaluated independently twice each by the same judge agent yielded 92% dimension-level exact agreement and 97% Solve agreement, providing direct evidence of instance-level repeatability under our structured rubric.


Question 2: Can you provide pass@k and pass^k for (k>1), with confidence intervals from repeated independent solver runs and evaluator variability measured separately?

Answer 2: As discussed in Response 1, historical runs did not consistently preserve pre-evaluation container snapshots, preventing faithful retroactive reconstruction of all historical submission states. We are conducting repeated independent runs of the solve agent to report pass@3 and pass^3 with 95% confidence intervals, and separately evaluate fixed outputs from the solve agent starting from independent clones of saved pre-evaluation snapshots to isolate the variability of the judge agent.


Question 3: What evidence supports low residual contamination across the evaluated model families beyond the single-model, no-issue-description screening procedure?

Answer 3: As discussed in Response 2, while our DeepSeek-V3.2 no-issue-description screening focuses on excluding clear memorization signals, SWE-Cycle provides a reusable construction methodology that can be directly applied to private or more recent datasets to mitigate contamination risks associated with previously published tasks.


Question 4: Under matched tasks, budgets, and code-and-test acceptance criteria, what statistically supported change in end-to-end success is attributable specifically to requiring environment reconstruction rather than providing a preconfigured environment?

Answer 4: As shown in Response 2, across paired runs in Appendix J, providing a preconfigured environment (−Env) increased end-to-end success from 12.35% to 20.37% for Claude-Sonnet-4.6 and from 13.55% to 19.30% for GLM-5.1, raising pooled success from 12.95% to 19.84%. Comparing full-requirement completion across the two settings (all six sub-dimensions including Env in FullCycle vs. the four Code and Test sub-dimensions in −Env) directly quantifies the additional end-to-end barrier imposed by autonomous environment delivery. Complementing these aggregate gains, the OpenLibrary and Matplotlib trajectory audits expose concrete failures in dependency provisioning and runtime delivery that preconfigured evaluations omit, demonstrating critical bottlenecks that prevent autonomous completion without external intervention.


We thank the reviewer again for the constructive comments, which help us clarify the contribution of SWE-Cycle and the evidence supporting its evaluation methodology.

Reviewer_cNxiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SWE-Cycle, a benchmark that evaluates coding agents across environment setup, implementation, and test generation. It also proposes SWE-Judge, which combines code review and execution-based checks. Experiments on 489 tasks show a large gap between isolated and end-to-end performance.

Critical strengths
  1. Important practical problem. Evaluating the full software development process is more realistic than only checking code patches.

  2. Interesting evaluation approach. SWE-Judge tries to address limitations of fixed test scripts, and the authors provide substantial experiments and human validation.

Critical weaknesses
  1. Unfair comparison between settings. The isolated and FullCycle tasks use somewhat different scoring criteria. For example, FullCycle can award the highest code dynamic score when "almost all" tests pass, while isolated implementation requires all relevant tests to pass. This makes the reported performance differences difficult to interpret.

  2. Questions about judge reliability. Human annotators had access to SWE-Judge's reasoning, which could introduce bias. The reported agreement rates alone are not enough to establish that the evaluator is consistently reliable.

Questions for authors
  1. Can you re-evaluate isolated and FullCycle results using exactly the same scoring criteria?

  2. Can you provide a human evaluation where annotators do not see SWE-Judge's scores or explanations?

3 comments

Dear Reviewer cNxi,

We thank you for recognizing the practical value of evaluating the full software development process and the contribution of SWE-Judge. Your comments identify two important concerns—consistency between the scoring criteria and the independence of human evaluation—which we address separately below.


Weakness 1: Unfair comparison between settings. The isolated and FullCycle tasks use somewhat different scoring criteria. For example, FullCycle can award the highest code dynamic score when "almost all" tests pass, while isolated implementation requires all relevant tests to pass. This makes the reported performance differences difficult to interpret.

Response 1: Thank you for catching this discrepancy. The phrase “(or almost all)” in the FullCycle rubric summary table (Appendix B.5, Table 11) was a wording error in the manuscript. In our actual evaluation prompts and grading execution, both isolated Code Implementation (Impl, Appendix B.3, Table 9) and FullCycle strictly require all relevant tests to pass to earn the highest Code Dynamic score (2/2), so all reported experimental results are already evaluated under identical scoring criteria. We will revise Table 11 and Appendix B.5 to make this consistency explicit.


Weakness 2: Questions about judge reliability. Human annotators had access to SWE-Judge's reasoning, which could introduce bias. The reported agreement rates alone are not enough to establish that the evaluator is consistently reliable.

Response 2: To eliminate potential anchoring bias from viewing rationales produced by the judge agent (Appendix E, Appendix H.1), we conducted an additional fully blind human evaluation of 300 submissions spanning all four task types (Env, Impl, TestGen, and FullCycle). Two new annotators who had not participated in the original audit independently scored all 300 submissions without access to SWE-Judge’s scores, explanations, reasoning, or final judgments, achieving 94.9% dimension-level exact agreement with each other, while a third independent annotator reviewed the submitted artifacts to adjudicate disagreements. Against the resulting adjudicated human labels, SWE-Judge achieved 94.3% dimension-level exact agreement and 98.3% Solve agreement. These blind-audit results provide direct, unbiased evidence of our judge agent’s reliability, and we will incorporate them into Section 4.2 and Appendix E alongside a clear distinction from the original human audit.


Question 1: Can you re-evaluate isolated and FullCycle results using exactly the same scoring criteria?

Answer 1: No re-evaluation is required because the actual evaluations of isolated Impl and FullCycle already enforced the exact same full-credit scoring criterion (requiring all relevant tests to pass for a 2/2 Code Dynamic score). As noted in Response 1, the discrepancy was solely a wording error in the manuscript’s summary rubric table (Appendix B.5, Table 11), rather than a difference in the executed evaluation criteria. All reported results already reflect identical scoring standards, and we will update the manuscript text and Table 11 accordingly.


Question 2: Can you provide a human evaluation where annotators do not see SWE-Judge's scores or explanations?

Answer 2: Yes. As detailed in Response 2, we completed an additional blind human evaluation of 300 submissions using two new annotators who had no access to SWE-Judge’s scores or explanations, followed by independent artifact adjudication by a third annotator. Against the adjudicated human labels, SWE-Judge achieved 94.3% dimension-level exact agreement and 98.3% Solve agreement.


We thank the reviewer again for the constructive comments, which help us clarify our scoring protocol and strengthen the evidence for the reliability of our judge agent.

Dear Reviewer cNxi,

I hope this message finds you well. As the discussion period is nearing its end with only about one day remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Dear Reviewer cNxi,

As the discussion period enters its final 6 hours, we wanted to briefly let you know that we have uploaded our revised manuscript (with all updates highlighted in blue) reflecting both points raised in your review:

Consistent Full-Credit Scoring Criteria: Corrected the summary wording in Table 11 and clarified the identical full-credit requirement across isolated Impl and FullCycle in Appendix B.5.

Independent Fully Blinded Human Evaluation: Added the 300-submission blinded human scoring protocol and results (94.3% dimension-level exact agreement and 98.3% Solve agreement) in Section 4.2, Appendix E, and Table 15.

We hope our responses and the updated manuscript have fully resolved your concerns regarding soundness and evaluation reliability. We would greatly appreciate any final thoughts or feedback you might share before the discussion window closes. Thank you again for your time and valuable insights!

Reviewer_MWAsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing issue-resolution benchmark for coding agent is mostly implemented in an isolated setting. As an example, it starts in a preconfigured environment, focusing only on patch generation, or correctness under certain test cases. Such setting may be limited in showing fully autonomous agent capability of end-to-end tasks. And also that script-based evaluation scoring can sometimes incorrectly judge the agent performance.

To solve these problems, the authors proposed SWE-cycle and SWE-judge. SWE-cycle is a benchmark for end-to-end issue resolution problem that evaluate three capabilities of agent, namely, environment reconstruction, patch implementation, and test generation. The benchmark sample is curated from SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual, which filtered through certain criteria, such as potential data contamination, task complexity, and test reliability. It consists of 489 task samples. The evaluation is run on 7 LLMs using OpenCode as the harness. The paper introduces two scoring metrics, Score and Solve, where score is the average of the static and dynamic testing, and Solve get scored if both static and dynamic test achieve maximum score. The result shows that end-to-end setting improve some upstream task from downstream information, as an example in the environment reconstruction task it get additional 10.8 percentage point and 7.4% for the Score and Solve metric respectively. However, on the broader view, this benefit is not consistent, in the final result of end-to-end setting the highest Solve metric only achieve 60.58% and, while there is no other model that achieve higher than 15% Solve rate except Claude-Fable-5. Showing the current coding agent (OpenCode) with several latest LLMs still hard to finish the task perfectly.

Critical strengths
  • The paper identifies an important gap between existing issue-resolution benchmarks and the broader capabilities required for autonomous software maintenance. The limitations of rigid script-based evaluation are also supported by concrete examples of incorrect assessments.
  • The authors put substantial efforts on validating SWE-Judge by comparing it with human annotation, and employing another LLM backbone to show the robustness.
  • The benchmark covers good numbers of curated samples (489) and evaluating comprehensively on seven LLM across both isolated and FullCycle settings. Also, the paired design and phase removal experiments provide useful insights on how different stages interact during end-to-end issue resolution.
Critical weaknesses
  • The technical section (Section 3) is thin. As a comprehensive benchmark, it is very useful to show the design consideration to fairly evaluate code agents, such as its comprehensiveness, the distribution of difficulty levels, etc. It is similar to setting an exam to students, the efforts, designs, and knowledge for setting exams are also useful in addition to the test itself. It helps to set the framework for continuous development of this field.
  • The paper claims to introduce the first benchmark covering the complete issue-resolution lifecycle. However, DevOps-Gym [ICLR26, Tang] already evaluates agents across interconnected DevOps stages, including build and configuration, issue resolution, and test generation, and includes end-to-end pipeline tasks. While SWE-Cycle differs in its focus, scale, and evaluation design, the paper does not show a clear comparison between their approach with DevOps-Gym. The authors should clarify what new capabilities or insights SWE-Cycle provides beyond this prior work.
  • In the isolated task, each task receive a 90 minute execution budget, where for FullCycle receives only 180 minutes for all three steps. It is unclear why FullCycle budget not set into 270 minute instead. Furthermore, the isolated tasks receive different level of information, eg., TestGen receives a reference-patched codebase while FullCycle must generate test based on its own implementation. These differences make it difficult to determine how much of the observed performance gap is cause specifically by task integration.
  • Alhough the paper provides substantial evidence of agreement with human evaluators and robustness across LLM backbone for SWE-judge. The SWE-judge agent directly read the artifacts and execution outputs that may introduce opportunities for evaluation manipulation, eg., agent-generated documents, scripts, or test outputs may potentially influence the judge agent. The paper does not establish whether SWE-Judge agent remains reliable under deliberately adversarial submission that may consist of prompt injection manipulating the agent.
  • All experiments only use OpenCode as the coding-agent harness. While this provides consistency among different LLMs, the failures result may be influenced by the harness architecture. Evaluating another agent harness would help determine whether these findings generalize to other coding-agent systems.
Questions for authors

How to define different levels of difficulties in tasks across software development life cycle?

How does this paper make sure the selected benchmark balances the levels of difficulties in the selected benchmark?

7 comments

Dear Reviewer MWAs,

We thank you for recognizing the importance of evaluating the complete issue-resolution lifecycle, the value of our paired design, and the contribution of SWE-Judge. Your comments help us clarify the benchmark’s design considerations, its relationship to prior work, and the scope of our experimental results. We address each concern below.


Weakness 1: The technical section (Section 3) is thin. As a comprehensive benchmark, it is very useful to show the design consideration to fairly evaluate code agents, such as its comprehensiveness, the distribution of difficulty levels, etc. It is similar to setting an exam to students, the efforts, designs, and knowledge for setting exams are also useful in addition to the test itself. It helps to set the framework for continuous development of this field.

Response 1: Thank you for this suggestion. Beyond releasing a dataset, our work introduces an evaluation paradigm for autonomous software development backed by a reusable task-construction and verification pipeline. This pipeline transforms tasks from existing benchmarks into full-lifecycle development tasks, supports constructing new tasks from high-quality GitHub pull requests, and verifies agent deliverables through SWE-Judge. Our technical contribution therefore encompasses both a curated benchmark and an extensible framework for continuous benchmark development and evaluation.

Specifically, our framework rests on two core design considerations. First, constructing paired Isolated and FullCycle tasks from the exact same issues (Section 3.2) enables us to systematically diagnose how isolated capabilities translate into full-lifecycle performance and examine cross-phase dependencies through phase-removal ablations (Section 4.5). Second, SWE-Judge combines static artifact inspection with dynamic execution under explicit task requirements and scoring rubrics (Section 3.3, Appendix B), extending evaluation beyond behaviors covered by fixed test scripts; its reliability is supported by both human audits (Section 4.2, Appendix E) and backbone comparisons for the judge agent (Appendix D). While we instantiate this methodology on SWE-bench Verified, Pro, and Multilingual, the same construction process readily extends to other repository-level tasks.

Regarding comprehensiveness and data quality, our instances are curated from 1,531 raw tasks across three established benchmarks and undergo three-stage filtering for potential contamination, lifecycle complexity, and test reliability (Section 3.1, Appendix A), retaining 489 instances across 9 programming languages. We further characterize task difficulty across the full lifecycle—jointly evaluating environment-configuration complexity, implementation scope and cross-module coupling, bug subtlety, and test-verification demands—under which easy, medium, and hard tasks account for 22.09%, 34.76%, and 43.15% of the 489 instances, respectively. We will expand Section 3 (Methodology) to present the complete design rationale, quality-control pipeline, and difficulty classification criteria.

Weakness 2: The paper claims to introduce the first benchmark covering the complete issue-resolution lifecycle. However, DevOps-Gym [ICLR26, Tang] already evaluates agents across interconnected DevOps stages, including build and configuration, issue resolution, and test generation, and includes end-to-end pipeline tasks. While SWE-Cycle differs in its focus, scale, and evaluation design, the paper does not show a clear comparison between their approach with DevOps-Gym. The authors should clarify what new capabilities or insights SWE-Cycle provides beyond this prior work.

Response 2: Thank you for pointing out this related work. Beyond covering the complete issue-resolution lifecycle, SWE-Cycle provides distinct diagnostic insights, evaluation reliability, and task-curation rigor compared with DevOps-Gym:

First, SWE-Cycle introduces a paired diagnostic design: the exact same 489 issues are evaluated under both Isolated (Env, Impl, TestGen) and FullCycle settings, enabling direct phase-level comparisons and cross-phase dependency analysis via phase-removal experiments (Section 4.5). Rather than measuring end-to-end completion alone, this design answers a distinct diagnostic question: how capabilities demonstrated in isolation transfer or degrade when a single solve agent must construct, preserve, and coordinate its own environment, implementation, and verification tests within one continuous session.

Second, SWE-Judge’s hybrid static–dynamic evaluation is substantially more reliable than deterministic script-based grading pipelines such as that in DevOps-Gym. In end-to-end workflows, our judge agent separates runtime execution gains from static structural degradation and test-hacking behaviors (Sections 4.3 and 4.5)—for example, writing custom verification tests when a solve agent’s self-generated tests omit critical scenarios such as account-deletion cleanup in NodeBB (Appendix C.1). Moreover, across 371 Script–Judge disagreements where human adjudication supports SWE-Judge in 98.6% of cases (Section 4.4, Appendix H), it avoids three systematic script failure modes: rejecting valid alternative implementations (e.g., Valkey #1499 and Teleport, Appendix H.6), misclassifying parser incompatibilities as agent failures (e.g., Gradle vs. Maven output formatting in Apache Lucene, Appendix H.7), and overlooking regressions outside target tests (e.g., Django #13590, Appendix H.8).

Third, SWE-Cycle can be systematically constructed atop established benchmarks and enforces a rigorous three-stage filtering pipeline—screening 1,531 candidate tasks from SWE-bench Verified, Pro, and Multilingual for potential data contamination, lifecycle complexity, and test reliability (Section 3.1, Appendix A)—thereby ensuring high task quality across 489 instances in 9 programming languages.

Weakness 3: In the isolated task, each task receive a 90 minute execution budget, where for FullCycle receives only 180 minutes for all three steps. It is unclear why FullCycle budget not set into 270 minute instead. Furthermore, the isolated tasks receive different level of information, eg., TestGen receives a reference-patched codebase while FullCycle must generate test based on its own implementation. These differences make it difficult to determine how much of the observed performance gap is cause specifically by task integration.

Response 3: Thank you for distinguishing the execution-budget and input-information differences between settings.

Regarding the execution budget: In the Isolated setting, all three tasks (Env, Impl, TestGen) start independently from scratch, each requiring the solve agent to explore and establish repository understanding anew. In contrast, FullCycle preserves a single continuous session state, allowing the solve agent to reuse repository understanding and intermediate work across phases (Section 4.1). This shared context motivated our 180-minute FullCycle budget rather than a 270-minute sum of three independent 90-minute budgets.

We examined execution records across 2,932 recoverable FullCycle runs spanning six of the evaluated models: the overall timeout rate of the solve agent was 2.18%, and the 95th-percentile execution time was 65.45 minutes, well below the 180-minute limit. These runtime statistics indicate that the 180-minute budget leaves ample headroom for the vast majority of runs, and the performance degradation in FullCycle stems primarily from limitations in model capabilities across long-horizon, cross-phase coordination rather than execution-time constraints.

Regarding input information: Supplying the reference-patched codebase in isolated TestGen (Section 3.2) is an intentional prerequisite to isolate post-implementation verification test generation, allowing us to assess whether a solve agent can design effective discriminative tests against a standardized upstream fix without conflating test generation with bug fixing. Conversely, FullCycle requires the solve agent to generate tests based on its own implementation, making the coupling between implementation and verification an intrinsic part of autonomous issue resolution. Generating tests from only the issue description and the unpatched codebase is also a valuable setting, and we will include it in subsequent experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Core Paradigm Contribution, Rebuttal Resolutions, and Discussion Status

Dear AC, SAC, and PCs,

Thank you and the reviewers for your guidance. All reviewers recognized the value of evaluating the full maintenance lifecycle, our paired Isolated–FullCycle design, and SWE-Judge's validation. We uploaded a revised manuscript (updates in blue). Since Reviewer cNxi and Reviewer MWAs have not yet responded, we summarize our core paradigm contribution and issue resolutions for your meta-review:

I. Core Contribution: A Generalizable Paired-Diagnostic Paradigm

We introduce a paired diagnostic evaluation paradigm and hybrid verification framework to move past assessing agents narrowly via isolated sub-tasks or coarse pass rates:

  1. Same-Issue Paired Diagnostics (Isolated != Autonomy): Evaluating the exact same 489 issues across Isolated (Env, Impl, TestGen) and single-session FullCycle settings (plus phase-removal ablations) exposes a key blind spot: strong isolated scores fail to yield end-to-end autonomy (FullCycle Solve drops below 14% for 6 of 7 frontier models).
  2. Hybrid Static–Dynamic Verification (SWE-Judge): Combining static review with dynamic execution corrects script misjudgments (human experts support SWE-Judge in 98.6% of 371 disagreements) and reveals a core trade-off: runtime feedback boosts upstream dynamic pass rates while accumulating static structural debt and inducing downstream test-hacking.
  3. Broad Generalizability: Pairing isolated checks with closed-loop execution under static–dynamic grading extends directly to new repositories and live PRs, offering a general blueprint for broader multi-stage agent domains to measure genuine autonomous capability rather than one-sided metrics.

II. Resolution of Reviewer Concerns

1. Reviewer cNxi (Rating: 2, Soundness: 1 — No Response Yet): 100% Resolved Both verifiable concerns underlying the soundness score are completely resolved:

  • Identical Criteria: “(or almost all)” in Table 11 was purely a summary typo. Actual prompts and grading for both isolated Impl and FullCycle strictly require all relevant tests to pass for full CODE_DYNAMIC credit (2/2). All reported results already use identical criteria (fixed in Table 11 & App. B.5).
  • New 300-Sample Blinded Human Eval: Two new independent annotators scored 300 submissions across all four tasks without seeing SWE-Judge outputs (94.9% inter-annotator exact agreement), adjudicated by a third annotator. SWE-Judge achieved 94.3% dimension exact agreement and 98.3% Solve agreement (Sec. 4.2, App. E.4, Table 15).

2. Reviewer MWAs (Rating: 3, Weak Accept — No Response Yet): All Points Addressed

  • DevOps-Gym: Added dedicated comparisons on paired transfer diagnostics, bare-repo setup, and script resilience (Secs. 1, 2, Table 1).
  • Methodology & Difficulty: Detailed our reusable pipeline, lifecycle difficulty split (22.09% easy, 34.76% medium, 43.15% hard), and post-fix isolation in TestGen (Secs. 3, 3.1, 3.2, App. A.3).
  • Budget, Harness & Security: Added FullCycle runtime stats across 7 models (medians: 5.7–29.5 min) and 2,932 runs showing a 2.18% timeout rate and 65.45-min P95 (ample headroom under 180 min), framed OpenCode as a controlled system variable (Sec. 4.1), and documented prompt isolation with zero injection attempts (App. B.1, Sec. 5).

3. Reviewer bCTj (Rating: 2 — Responded Nov 16): Progress & Context Reviewer bCTj acknowledged our contamination controls, 300-sample blinded eval, and judge repeatability study (100 double-evals: 92% dimension exact, 97% Solve agreement). On the two remaining points:

  • Multi-Run pass@k: Re-running >13,600 long-horizon containerized trajectories (up to 180 min/run) is infeasible within the short rebuttal window. Following SWE-bench Verified/Pro practice, we report single-run results with 95% bootstrap/Wilson CIs (App. F), backed by cross-backbone (<3% Solve diff, App. D) and same-backbone (97% Solve) judge stability.
  • Value of Env: Beyond lowering full completion from 19.84% (-Env) to 12.95% (FullCycle), Table 28 (App. J) evaluates identical Code/Test criteria and shows omitting Env directly shifts Code by -0.32 (-8.0%), alongside concrete runtime-delivery failures in OpenLibrary, Matplotlib, Sphinx, and Ansible (App. F.1).

Thank you for your time and consideration.

Best regards,

Authors of Submission 12922

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.