- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-Cycle, a benchmark of 489 issues curated from SWE-bench Verified, Pro, and Multilingual. It evaluates environment reconstruction, code implementation, and verification test generation both independently and within a single FullCycle agent session. The paper also proposes SWE-Judge, an LLM-driven evaluator agent that combines static artifact inspection with dynamic execution and reference-assisted test refinement. Experiments with seven solver models examine task performance, differences between isolated and integrated execution, and cross-phase dependencies. Validation includes human auditing, a comparison of two evaluator backbones, and phase-removal ablations. The reported results highlight a gap between high average scores and perfect completion across all lifecycle dimensions.
Critical strengths
S1. A practically relevant evaluation scope with a useful paired design.
Requiring an agent to produce its own environment, implementation, and verification tests addresses aspects of repository maintenance that patch-only scoring does not explicitly assess. Evaluating the same issues in isolated and FullCycle settings provides a structured basis for examining how these capabilities interact, making the benchmark potentially useful for diagnosing failures beyond final patch correctness.
S2. An execution-grounded evaluator supported by substantial initial validation.
SWE-Judge supplements artifact inspection with actual execution and reference-assisted verification rather than relying exclusively on textual judgments. The 946 human inspections, two-backbone evaluator comparison, and documented failure cases provide meaningful initial evidence for its usefulness in identifying limitations of rigid script-based evaluation, although they do not fully resolve the reliability concerns below.
Critical weaknesses
W1. The reliability of the LLM-driven agentic evaluator and repeated solver execution is not sufficiently established.
SWE-Judge is an LLM-driven evaluator agent that autonomously inspects artifacts, invokes tools, executes tests, and may generate additional verification scripts. Its judgments can therefore depend on both the backbone model and the agent’s evidence-gathering and execution trajectory. Appendix D compares Claude-Opus-4.5 and GPT-5.4 on 414 evaluation trials, but does not establish the stability of repeated evaluations of identical submission snapshots by the same evaluator agent. Sensitivity to evaluator capability tiers and model-version changes is also insufficiently characterized, and similar aggregate solve rates can conceal substantial instance-level disagreement.
Separately, although Appendix F reports 95% confidence intervals, the manuscript does not clearly document repeated independent solver runs or separate solver variability from evaluator variability. Neither pass@k nor pass^k is reported for . These omissions limit the evidence for repeatable measurement and consistently successful autonomous execution, both of which are central to the paper’s claims.
W2. The incremental measurement contribution and robustness to data contamination require stronger evidence.
DeepSWE already evaluates autonomous repository-level engineering and analyzes agent self-verification. Explicit environment reconstruction is a potentially meaningful extension, but its independent effect on task completion remains unclear. The −Env ablation in Appendix J reports changes in remaining-phase scores rather than success rates under a common acceptance criterion, while also acknowledging changes in context, available artifacts, and effective budget. It therefore does not establish whether environment reconstruction introduces a substantial, statistically supported obstacle to completion or which additional failures it reveals beyond preconfigured evaluations.
Furthermore, all issues are inherited from existing benchmarks, while contamination screening relies on DeepSeek-V3.2 attempting tasks without their issue descriptions. Failure under this restricted setting does not establish that a model will not recall a solution once the issue is provided, nor that the screening result generalizes across model families. The uncharacterized residual contamination further limits confidence that the benchmark measures previously unseen problem-solving rather than prior exposure.
Questions for authors
-
How stable are instance-level scores and Solve decisions when identical submission snapshots are repeatedly assessed by the same LLM-driven evaluator agent and by evaluator agents using backbones from different capability tiers and model versions?
-
Can you provide pass@k and pass^k for , with confidence intervals from repeated independent solver runs and evaluator variability measured separately?
-
What evidence supports low residual contamination across the evaluated model families beyond the single-model, no-issue-description screening procedure?
-
Under matched tasks, budgets, and code-and-test acceptance criteria, what statistically supported change in end-to-end success is attributable specifically to requiring environment reconstruction rather than providing a preconfigured environment?
3 comments
Weakness 2: The incremental measurement contribution and robustness to data contamination require stronger evidence.
DeepSWE already evaluates autonomous repository-level engineering and analyzes agent self-verification. Explicit environment reconstruction is a potentially meaningful extension, but its independent effect on task completion remains unclear. The −Env ablation in Appendix J reports changes in remaining-phase scores rather than success rates under a common acceptance criterion, while also acknowledging changes in context, available artifacts, and effective budget. It therefore does not establish whether environment reconstruction introduces a substantial, statistically supported obstacle to completion or which additional failures it reveals beyond preconfigured evaluations.
Furthermore, all issues are inherited from existing benchmarks, while contamination screening relies on DeepSeek-V3.2 attempting tasks without their issue descriptions. Failure under this restricted setting does not establish that a model will not recall a solution once the issue is provided, nor that the screening result generalizes across model families. The uncharacterized residual contamination further limits confidence that the benchmark measures previously unseen problem-solving rather than prior exposure.
Response 2: Thank you for asking us to clarify our incremental measurement contribution and contamination controls. SWE-Cycle’s contribution extends beyond adding lifecycle stages: it constructs paired Isolated and FullCycle settings for the exact same 489 issues, compares phase-level performance across settings, and diagnoses cross-phase dependencies through phase-removal ablations (Section 4.5, Appendix J). Together with the dataset, SWE-Cycle provides a reusable evaluation methodology that can extend to broader benchmarks to systematically diagnose how isolated capabilities translate into full-lifecycle autonomy.
Effect of environment reconstruction. The table below compares end-to-end success (Solve) rates across paired runs in Appendix J between FullCycle (requiring full credit across all six Env, Code, and Test Static/Dynamic sub-dimensions) and −Env (requiring full credit across the four remaining Code and Test sub-dimensions):
| Solve Agent | FullCycle success rate | −Env success rate |
|---|---|---|
| Claude-Sonnet-4.6 | 12.35% | 20.37% |
| GLM-5.1 | 13.55% | 19.30% |
| Pooled | 12.95% | 19.84% |
Success rates are consistently higher under −Env, showing that satisfying full-lifecycle requirements—including environment reconstruction—poses a substantial additional obstacle to autonomous completion. Inspecting paired evaluation trajectories reveals concrete environment-delivery failures that preconfigured evaluations miss:
- Incomplete dependency setup (OpenLibrary, Claude-Sonnet-4.6, instance containing
ba3abfb6): The FullCycle submission’s setup script relied on a nonexistent Miniconda installation, while the system Python environment lacked required packages includingrequestsandpytest. The judge agent had to install missing dependencies and remove conflicting pytest plugins before verification could proceed. - Unusable designated runtime (Matplotlib, Claude-Sonnet-4.6,
matplotlib__matplotlib-23412): Starting Python in the designatedtestbedenvironment returnedInvalid argument, whereas the required packages were installed in thebaseenvironment. The judge agent had to fall back tobaseto import the project and collect tests; thus, although functional tests could run in a fallback environment, the submission failed to deliver the required working runtime.
These cases demonstrate that SWE-Cycle additionally assesses whether a solve agent can deliver a usable designated runtime, install required dependencies, and leave behind a verifiable execution state—capabilities omitted when evaluating code and tests solely inside preconfigured containers, yet essential for autonomous completion without external intervention.
Contamination screening. Because pretraining corpora across evaluated model families are undisclosed, we mitigate memorization risks at both the task-filtering and task-formulation levels. First, our no-issue-description screening with DeepSeek-V3.2 removes 128 instances exhibiting direct memorization signals (Section 3.1, Appendix A). Second, FullCycle requires the solve agent to autonomously configure a bare environment, implement the fix, and construct discriminative verification tests under joint static–dynamic grading—tasks that cannot be passed merely by recalling an isolated code patch. Moreover, because SWE-Cycle provides an extensible task-construction methodology alongside the dataset, its pipeline can be directly applied to private repositories or newly collected pull requests to continuously refresh evaluation sets.
Question 1: How stable are instance-level scores and Solve decisions when identical submission snapshots are repeatedly assessed by the same LLM-driven evaluator agent and by evaluator agents using backbones from different capability tiers and model versions?
Answer 1: As discussed in Response 1, our initial study of 100 submissions evaluated independently twice each by the same judge agent yielded 92% dimension-level exact agreement and 97% Solve agreement, providing direct evidence of instance-level repeatability under our structured rubric.
Question 2: Can you provide pass@k and pass^k for (k>1), with confidence intervals from repeated independent solver runs and evaluator variability measured separately?
Answer 2: As discussed in Response 1, historical runs did not consistently preserve pre-evaluation container snapshots, preventing faithful retroactive reconstruction of all historical submission states. We are conducting repeated independent runs of the solve agent to report pass@3 and pass^3 with 95% confidence intervals, and separately evaluate fixed outputs from the solve agent starting from independent clones of saved pre-evaluation snapshots to isolate the variability of the judge agent.
Question 3: What evidence supports low residual contamination across the evaluated model families beyond the single-model, no-issue-description screening procedure?
Answer 3: As discussed in Response 2, while our DeepSeek-V3.2 no-issue-description screening focuses on excluding clear memorization signals, SWE-Cycle provides a reusable construction methodology that can be directly applied to private or more recent datasets to mitigate contamination risks associated with previously published tasks.
Question 4: Under matched tasks, budgets, and code-and-test acceptance criteria, what statistically supported change in end-to-end success is attributable specifically to requiring environment reconstruction rather than providing a preconfigured environment?
Answer 4: As shown in Response 2, across paired runs in Appendix J, providing a preconfigured environment (−Env) increased end-to-end success from 12.35% to 20.37% for Claude-Sonnet-4.6 and from 13.55% to 19.30% for GLM-5.1, raising pooled success from 12.95% to 19.84%. Comparing full-requirement completion across the two settings (all six sub-dimensions including Env in FullCycle vs. the four Code and Test sub-dimensions in −Env) directly quantifies the additional end-to-end barrier imposed by autonomous environment delivery. Complementing these aggregate gains, the OpenLibrary and Matplotlib trajectory audits expose concrete failures in dependency provisioning and runtime delivery that preconfigured evaluations omit, demonstrating critical bottlenecks that prevent autonomous completion without external intervention.
We thank the reviewer again for the constructive comments, which help us clarify the contribution of SWE-Cycle and the evidence supporting its evaluation methodology.
Dear Reviewer bCTj,
We thank you for recognizing the value of our paired task design and execution-grounded evaluation. Your comments help us clarify measurement repeatability, the incremental contribution of SWE-Cycle, and the evidence concerning data contamination. We address each concern below.
Weakness 1: The reliability of the LLM-driven agentic evaluator and repeated solver execution is not sufficiently established.
SWE-Judge is an LLM-driven evaluator agent that autonomously inspects artifacts, invokes tools, executes tests, and may generate additional verification scripts. Its judgments can therefore depend on both the backbone model and the agent’s evidence-gathering and execution trajectory. Appendix D compares Claude-Opus-4.5 and GPT-5.4 on 414 evaluation trials, but does not establish the stability of repeated evaluations of identical submission snapshots by the same evaluator agent. Sensitivity to evaluator capability tiers and model-version changes is also insufficiently characterized, and similar aggregate solve rates can conceal substantial instance-level disagreement.
Separately, although Appendix F reports 95% confidence intervals, the manuscript does not clearly document repeated independent solver runs or separate solver variability from evaluator variability. Neither pass@k nor pass^k is reported for (k>1). These omissions limit the evidence for repeatable measurement and consistently successful autonomous execution, both of which are central to the paper’s claims.
Response 1: Thank you for identifying these important questions concerning evaluation repeatability and variance across runs of the solve agent. We summarize our available evidence and concrete plans for expanded variance measurement below.
First, SWE-Judge operates under explicit, dimension-specific 0–2 scoring rubrics (Appendix B) that specify exact Static and Dynamic assessment targets, score thresholds, and deterministic rules for handling conditions such as environment failures and vacuous tests, minimizing discretionary interpretation. In an initial repeatability study involving 100 submissions evaluated independently twice each by the same judge agent, we observed 92% dimension-level exact agreement and 97% Solve agreement between evaluations, providing direct initial evidence of instance-level repeatability under our rubric.
Second, we conducted an independent blind human evaluation of 300 submissions covering all four task types (Env, Impl, TestGen, and FullCycle), where two annotators scored submissions independently without seeing SWE-Judge outputs and a third annotator adjudicated disagreements against the raw artifacts. SWE-Judge achieved 94.3% dimension-level exact agreement and 98.3% Solve agreement with the adjudicated human labels, providing complementary instance-level evidence of judgment accuracy.
Finally, the judge agent executes inside the solve agent’s container to inherit its live environment and artifacts (Section 4.1). To explicitly separate the variability of the solve agent from that of the judge agent across multi-run trials, we are archiving pre-evaluation container snapshots to launch repeated evaluations from independent clones on fixed outputs from the solve agent, while conducting repeated independent runs of the solve agent to report
pass@3andpass^3in the revised manuscript.