acceptodds
Under review as a conference paper at ICLR 2027

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

Abstract

Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as bug fixing. However, in the real world, software engineering is an inherently iterative process driven by evolving requirements and long-term feature development. Static, one-shot evaluation paradigms fail to capture this dynamic. To bridge this gap, we propose SWE-CI, a novel repository-level benchmark grounded in the Continuous Integration loop, designed to shift the evaluation paradigm for coding agents from static, short-term functional correctness toward dynamic, long-term regression-aware code evolution. The key insight is simple: A testable aspect of maintainability can be revealed by tracking how functional correctness changes across successive code modifications. The benchmark comprises 100 tasks, each derived from a real-world code repository with a development history spanning an average of 233 days and 71 consecutive commits. SWE-CI requires agents to systematically resolve these tasks through multiple rounds of analysis and coding iterations. Extensive experiments shed light on regression, a problem systematically overlooked by existing benchmarks: an acceptable final deliverable does not imply an acceptable development process.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Coding, Maintainability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gWVzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SWE-CI, a benchmark designed to evaluate the ability of coding agents to continuously maintain codebases over the long term. The benchmark constructs 100 tasks based on real GitHub repository histories, utilizing an Architect–Programmer dual-agent CI loop to iteratively modify and correct code. Additionally, it proposes an EvoScore to track correctness changes throughout the development process. Finally, the paper evaluates a broad range of model families and finds that they often suffer from severe functional regressions during development.

Critical strengths
  1. Valuable evaluation setting and comprehensive experiments. This paper proposes analyzing the entire trajectory across the modification history of the codebase, and provides an in-depth discussion on intermediate regressions that the final success rate cannot capture. The comprehensive experiments across a wide range of model families are also meaningful for drawing general conclusions and would benefit the SWE community.
Critical weaknesses
  1. Limited novelty due to missing discussion with closely related work. MaintainCoder [1] and SWE-Milestone [2] have already focused on evaluating agents' abilities to maintain dynamic codebases, with a particular focus on regression and evolution during this process. This paper does not clearly discuss these works, which weakens its novelty and contribution.

  2. The design remains oversimplified compared to real-world code maintenance. When resolving long-range commit histories, the agent only has access to the start and target versions without replaying real-world step-by-step development. Furthermore, the agent has access to the target version's test cases, making it more of an iterative development setting rather than long-term codebase maintenance. Additionally, the paper requires that dependencies cannot change between the start and target versions, which also diverges from long-term real-world SWE scenarios.

[1] Wang, Zhengren, et al. "Maintaincoder: Maintainable code generation under dynamic requirements." Advances in Neural Information Processing Systems 38 (2026): 15399-15436.

[2] Deng, Gangda, et al. "SWE-Milestone: Evaluating AI agents on continuous software evolution." International Conference on Machine Learning. PMLR, 2026.

Questions for authors
  1. What are the core differences between the proposed benchmark and existing work such as MaintainCoder and SWE-Milestone? Could the authors provide a more systematic comparison?

  2. SWE-CI evaluates code evolution toward a fixed target version and excludes commit spans with dependency changes. How do the authors justify the benchmark's representativeness in reflecting realistic, long-term software evolution?

3 comments

Thank you for your review, and for recognizing the value of trajectory-level evaluation and the scale of our experiments. Our responses are below.

W1 / Q1: Differences from MaintainCoder and SWE-Milestone

Thank you for pointing out these two related works. The revision will discuss and cite them in the related work section. Here is a comparison of the three:

MaintainCoder (MaintainBench)SWE-MilestoneSWE-CI
GranularityProblem-level: 500+ Python problems, extended from 5 datasets such as HumanEval, MBPP and APPSRepository-level: 7 repositories, 5 languagesRepository-level: 67 repositories, Python
Source of requirementsOne requirement change per problem, generated by GPT-4o and then refinedA milestone dependency graph rebuilt from commit history, with one requirement document per milestoneRegenerated in each round by the Architect from the current test failures of the codebase
Length of evolutionOne modification14 milestones per repository on averageUp to 20 rounds, 13.9 on average (Table 2)
How regressions are reflectedPost-change Pass@k, amount of code change, AST similarityF1 per milestone; broken previously passing tests count against precisionTracked every round; we report regression frequency, magnitude, trend over rounds, and recovery

The core differences:

  • MaintainCoder: from one modification to continuous evolution. MaintainCoder defines dynamic maintainability as the cumulative maintenance effort over a sequence of requirement changes. To reduce cost and variance, its evaluation uses a first-order truncation and assesses only one change. SWE-CI evaluates this sequence itself, on real repositories: the same codebase is modified continuously for up to 20 rounds. MaintainCoder also finds that static metrics contradict each other, while its dynamic metrics, measured after requirement changes, are highly consistent. This supports evaluating through continuous modification, as SWE-CI does.
  • SWE-Milestone: who decides the requirements, and what is evaluated. Both benchmarks modify the same codebase continuously, and both account for regressions. They differ in three ways:
    1. SWE-Milestone's requirements are decomposed into milestones from history in advance, and they do not change with the agent's behavior. In SWE-CI, each round's requirements are determined by the current failures of the codebase, so regressions introduced by the agent itself become later requirements. SWE-CI therefore evaluates whether an agent can drive evolution on its own without breaking existing functionality.
    2. SWE-Milestone scores each milestone. SWE-CI records the codebase state at every round and analyzes how regressions accumulate and recover over rounds (§4 Q2–Q4).
    3. SWE-CI covers 67 repositories and 31 models. SWE-Milestone covers 7 repositories and 12 models, but includes 5 languages.

W2 / Q2: Simplifications of the setting, and representativeness

We agree that SWE-CI does not cover every form of long-term evolution. It represents the most common one: continuously extending and modifying an existing codebase in a stable environment. We believe the following simplifications do not affect its representativeness for this setting:

  • Not replaying intermediate historical steps. This is a deliberate design choice. In real maintenance, what to do next is decided by the current state of the codebase, not by a pre-written path. In each round, SWE-CI generates at most five requirements from the current failures, and across 31 models each task takes 13.9 CI iterations on average (Table 2). Replaying history step by step evaluates the ability to follow a known path. SWE-CI evaluates the ability to keep existing functionality intact while making progress on one's own. The two are complementary.
  • Access to the target version's tests. In CI, the test suite is part of the codebase. When a pipeline fails, developers read the failing tests to locate the problem. Under ISO/IEC/IEEE 14764, modifications made after delivery to add or improve functionality are themselves software maintenance (the enhancement category). Moreover, seeing the tests does not make the task easy. Tracking individual tests, 28.19% of rounds break a previously passing test on average across 31 models, and only 16.54% of tasks are free of regressions throughout. The difficulty is not knowing the goal, but avoiding breaking existing functionality under continuous modification.
  • A fixed target version. This gives a verifiable end point. Every executable repository-level benchmark needs such an end point; in SWE-Milestone, each milestone is also verified against a historical version.
  • Fixed dependencies. This keeps the whole trajectory in one environment, so test failures can only come from the agent's code, not from changes in the environment. Among all maximal commit spans with unchanged dependencies, we selected the 100 with the longest time span and the most commits. On average, these tasks span 233 days and 71 commits, and involve 29.8 files and 1,592.7 changed lines (Table 1).

Dependency upgrades and changing external requirements are outside the scope of SWE-CI. The revision will state this in a new Limitations section.

We hope these responses answer your questions. We are happy to discuss further if anything remains unclear.

Thank you to the authors for the detailed responses and additional experiments. I will maintain my positive score.

Reviewer_TyJYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SWE-CI, a novel repository-level benchmark designed to evaluate the ability of LLM-powered agents to maintain codebases through iterative, regression-aware code evolution. The central motivation is well-grounded: existing benchmarks such as SWE-bench and HumanEval adopt a snapshot-based evaluation paradigm that grades agents solely on the functional correctness of their final output, systematically ignoring the quality of the development process. SWE-CI addresses this blind spot by shifting the evaluation paradigm from static, one-shot task completion to dynamic, multi-round code evolution grounded in the Continuous Integration loop. The benchmark comprises 100 tasks derived from real-world Python repositories, each defined by base and target commits spanning an average of 233 days and 71 consecutive commits of authentic development history. Agents iteratively modify the base codebase to pass the target test suite. To quantify performance across the full evolution trajectory, the paper introduces EvoScore, a time-weighted aggregate of per-round normalized correctness changes. Extensive experiments across 31 models from 9 providers reveal that functionally correct final outputs frequently conceal significant regressions throughout the development process.

Critical strengths
  • The shift from snapshot to evolution-based evaluation is well-motivated and clearly executed. Grounding tasks in real commit histories rather than synthetic scenarios adds credibility, and the CI loop structure gives the evaluation a concrete operational meaning.

  • EvoScore is simple and effective. It scores the full trajectory rather than just the final state, normalizes across tasks with different test counts, and stays stable across a wide range of gamma values (Kendall tau of 0.90 between gamma=0.8 and gamma=1.0). The case studies in Appendix D show clearly that two agents can both resolve a task yet score very differently, which is something resolution rate cannot capture.

  • The experimental scope is a practical strength. Results across 31 models show that stronger models differ from weaker ones mainly in how often they regress, not how severely, and that LLM-generated code scores better on Pylint but worse on the Maintainability Index across all models. These findings give concrete and useful insight into where current LLM coding behavior falls short.

Critical weaknesses
  • Oracle test access weakens the evaluation of requirement discovery. Appendices E.1 and E.2 describe the Architect's and Programmer's access to test source code. This provides information about expected behavior that may not be available from ordinary CI failure signals. The benchmark may therefore measure how well the Architect derives requirements from tests rather than discovers intended behavior from incomplete evidence. This weakens the claim that the dual-agent protocol independently evaluates requirement analysis.

  • The same target tests guide development and assess success, without a held-out assessment to check whether the implementations generalize beyond the visible tests. Passing these tests alone provides limited evidence that the agents implement the intended behavior rather than fit the available assertions and expected outputs.

  • EvoScore tracks the number of passing tests rather than which tests pass. An iteration that breaks previously passing tests while fixing an equal number of failing tests would be classified as stable. Similarly, recovering the previous passing-test count does not necessarily mean that the broken functionality has been restored. This can conceal regressions, which are central to the benchmark’s assessment.

  • Equation 2 uses different normalization references for improvements and regressions, so equal-sized changes in opposite directions do not contribute symmetrically. The paper does not assess sensitivity to this normalization or correlate EvoScore with independent quality measures or per-regression severity. It is therefore unclear how well the score reflects the practical consequences of regressions.

Questions for authors
  1. Are model rankings stable across repeated runs and different iteration budgets? Can the paper report variance estimates to show whether differences between closely ranked models are reliable, and assess sensitivity to the 20-iteration limit, particularly for tasks with larger test gaps or models that use more iterations?

  2. Do the role-separation findings hold for cross-provider combinations, such as Claude as Architect and GPT as Programmer? How do these compare with the existing same-family single-agent baseline in Table 5?

  3. All 67 repositories are public GitHub projects, and several are widely used and may be present in pretraining data. How does performance vary with repository popularity and target-commit age? Could prior exposure to public repository code affect the results?

3 comments

Thank you for your detailed review, and for recognizing the evaluation paradigm, EvoScore, and the scale of our experiments. Our point-by-point responses are below.

W1: The Architect can read test source code, which weakens the evaluation of requirement discovery

We adopted this design because reading test source code reflects real CI practice: test code usually lives in the repository, and when a pipeline fails, developers typically open the failing tests to inspect the assertions. In SWE-CI, the Architect must produce high-level requirements in natural language, not a restatement of the error messages. To test this, we ran a requirement-driven ablation. We removed all test files from the Programmer's environment. The Architect had to write self-contained requirement specifications, including interface signatures, edge cases, and input/output examples, and was not allowed to mention any test, test path, or nodeid. The requirement document was therefore the Programmer's only source of information about expected behavior. EvoScore (γ = 1) on 3 models:

ModelDefault (Table 2)Requirement-driven (no tests for Programmer)Change
GLM-50.7090.730+0.021
Qwen3.6-plus0.7160.701−0.015
Kimi-k2.50.4700.484+0.014

All changes are within 0.021, with no consistent direction. For reference, changing only the agent harness changes EvoScore by up to 0.029 (Table 4). GLM-5 and Qwen3.6-plus originally differ by only 0.007, and their order swaps in the ablation, which is within this range of variation. In other words, requirement analysis in the dual-agent protocol is substantive, not a simple restatement of the tests. Nevertheless, we will clarify in the paper that the dual-agent protocol separately evaluates requirement analysis based on CI feedback and implementation, rather than discovering expected behavior from vague external requirements.

W2: The same tests guide development and assess success, without held-out tests

In CI, using the same tests both to guide development and to decide whether a change can be merged is standard practice, because the tests are the project's definition of expected behavior. The target tests in SWE-CI are upstream tests written by the project developers for the target version, not tests generated for the benchmark. In addition, two factors reduce the risk of fitting the visible assertions:

  • Agents cannot modify the tests or run them on their own. They only see the external CI results after each round (Appendix E), so they cannot game the assertions through repeated trial runs.
  • In the requirement-driven ablation in W1, the Programmer, who writes the code, cannot see any test and implements only from natural-language requirements. EvoScore remains essentially unchanged. This shows that the implementations do not rely on fitting the visible assertions.

More importantly, we believe held-out tests may not suit the CI setting. Held-out tests presuppose a written specification that is independent of the tests (e.g., the instruction file in NL2RepoBench). The tests are then a sample of the specification, behavior beyond the visible tests can be derived from the specification, and the held-out part can test generalization. CI has no such specification; development is driven by the tests themselves. Randomly holding out some upstream tests would assume that the values of the hidden assertions can be derived from the visible assertions and values, and this is not guaranteed. For example, upstream tests often depend on function names, parameters, or exact return values that the requirements do not state. When a held-out test fails, we cannot tell whether the implementation overfit the visible assertions, or whether the held-out assertion simply could not be derived from the visible information.

W3: EvoScore tracks the number of passing tests, which can conceal regressions

This can happen in principle. When we recount by individual tests, there are indeed more regressions than counting shows, but the paper's conclusions do not change. A round counts as a regression if any test that passed in the previous round fails in this round. Means over 31 models:

MetricBy pass count (Table 2)By individual tests
Regression frequency17.09%28.19%
Zero-Reg28.77%16.54%
Recovery rate71.78%83.76%

Among rounds classified as "stable" or "progress" by pass count, 13.6% actually broke a previously passing test. The conclusion that stronger models regress less still holds. The Kendall τ between model rankings under the two definitions is 0.83 for regression frequency and 0.76 for Zero-Reg. Strong models (Res. Rate > 0.70) have a mean regression frequency of 18.26%, versus 39.80% for weak models (Res. Rate < 0.30).

On recovery: you noted that restoring the previous pass count does not mean the broken functionality has been restored. Under the individual-test definition, a regression counts as recovered only if all the tests it broke pass again in some later round. Under this more direct definition, the recovery rate is 83.76%, higher than the 71.78% based on pass count. This is because each regression is judged only by the tests it broke; tests broken later count as new regressions and no longer affect whether the earlier regression is recovered. Therefore, the conclusion that most regressions are eventually recovered (§4 Q4) also holds under the individual-test definition.

W4: Asymmetric normalization in Equation 2, and no comparison with independent quality measures

First, the asymmetry only arises when the codebase falls below its initial state. As long as n(c) ≥ n(c0), both progress and regression are normalized by the same gap, and equal-sized changes in opposite directions contribute exactly the same.

Using a different reference below the initial state is a deliberate design choice. Progress is measured relative to the gap to be closed, where a = 1 means the goal is reached. Regression is measured relative to the functionality that existed at the start, where a = −1 means all originally passing tests fail. This keeps a within [−1, 1], so tasks with different test counts can be averaged with equal weight. If both branches were normalized by the gap, regression would have no lower bound. For example, in Case 1 of Appendix D, the base passes 392 tests and the gap is only 140, so a single crash would give a = −2.8. The larger the base relative to the gap, the heavier a single crash would weigh, and a few tasks would dominate the average.

To check whether this design affects our conclusions, we computed a symmetric version: both branches are divided by the gap and clipped at −1. Its Kendall τ with the original EvoScore ranking is 0.82.

Comparison with an independent quality measure: we selected 200 pairs in which two models both resolved the same task and one had a higher EvoScore. We told an LLM judge that both solutions are functionally correct, and asked it to compare their maintainability in terms of modularity, readability, extensibility and code style. The two judges (GLM-5.2 and Qwen3.7-Plus) preferred the higher-EvoScore solution in 137 (68.5%) and 131 (65.5%) pairs, and agreed with each other on 156 pairs. We will add this experiment to the appendix.

Regression severity is measured separately: Table 2 reports regression magnitude, and Fig. 6 distinguishes crash from non-crash regressions and reports the recovery rate of each.

Q1: Variance across repeated runs, and sensitivity to the iteration budget

Due to cost (a single full evaluation of one model takes hundreds of millions of input tokens and dozens of hours, Table 10), each model in Table 2 was run once. To estimate run-to-run variation, we ran GLM-5 two more times under the same setting:

ModelRun 1 (Table 2)Run 2Run 3MeanStd. dev.
GLM-50.7090.6850.7170.7040.017

The standard deviation over three runs is 0.017, and the range is 0.032. The change of GLM-5 in the W1 ablation (+0.021) is comparable to this variation. The two existing perturbation experiments also indicate the noise level: changing the agent harness changes EvoScore by up to 0.029 (Table 4), and switching to the requirement-driven setting, where the Programmer cannot see tests, changes it by up to 0.021 (see W1). Among the 30 pairs of adjacently ranked models in Table 2, 10 differ by more than 0.035, which exceeds these variations. Models with very small differences should therefore be regarded as the same tier, and the revision will mark this in Table 2.

On the iteration budget: 20 rounds is a deliberate choice, intended to give a budget that is sufficient while keeping cost manageable. Under this budget, more rounds are unlikely to change the conclusions:

  • Strong models do not need 20 rounds. The 6 models in Table 2 with a Res. Rate of at least 84% use only 4.45–9.55 rounds per task on average.
  • Trajectories that run all 20 rounds largely plateau in later rounds. They are usually stuck on a problem that is never fixed. For example, in Case 2 of Appendix D, Qwen3.5-plus stays at 62–63 passing tests (out of a target of 65) from round 10 onward, with no further progress over the last 10 rounds. Across all unresolved trajectories, the best pass count after round 15 improves by only about 6% of the test gap on average.

On the other hand, each additional round requires another full Architect and Programmer call and another pytest run of up to one hour (Table 10). Raising the budget would therefore mainly increase cost without materially changing the results.

Q2: Cross-provider role combinations

We added cross combinations of Claude Opus 4.7 and GPT-5.4 (EvoScore, γ = 1):

Architect \ ProgrammerClaude Opus 4.7GPT-5.4
Claude Opus 4.70.8720.677
GPT-5.40.8540.582

The diagonal entries are the results from Table 2, where the same model plays both roles. The findings are consistent with the GLM family. With the Architect fixed, changing the Programmer changes EvoScore by 0.195–0.272. With the Programmer fixed, changing the Architect changes it by 0.018–0.095. Changing the Programmer therefore has the larger effect. Whichever role is fixed, the order of the two models stays the same. We also see no extra advantage from pairing a model with itself: with Opus 4.7 as the Architect, GPT-5.4's score rises from 0.582 to 0.677. The single-agent baseline has so far been run only on the GLM family (Table 5), where it is 0.028–0.051 lower than the dual-agent setup with the same model.

Q3: Repository popularity, commit age, and pretraining exposure

We examined how task performance relates to these two factors. The Spearman correlation between each task's mean EvoScore (over 31 models) and repository stars is only 0.27, and the correlation with the target commit time is only 0.35. Moreover, memorizing code snapshots helps little in SWE-CI: agents must modify the code step by step under requirements that are generated dynamically in each round, and the ability to avoid regressions cannot be acquired through memorization.

We hope these responses address your concerns. If so, we would be grateful if you would consider raising your score.

Reviewer_X1ZaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWE-CI, a benchmark for evaluating how well AI coding agents maintain and improve codebases through repeated development cycles. The authors argue that evaluating only the final solution overlooks an important problem: an agent may eventually pass all tests while repeatedly breaking existing functionality along the way. SWE-CI includes 100 tasks from 67 Python repositories, representing an average of 233 days and 71 commits of development history. Each task follows a continuous integration loop in which an Architect agent analyzes test failures and generates incremental requirements, while a Programmer agent implements the changes. The paper introduces EvoScore to measure functional progress across the entire sequence of changes, rewarding sustained improvement and penalizing regressions. Experiments across 31 models show that stronger models generally perform better because they introduce regressions less frequently, although even the best models struggle to maintain consistently stable development. The authors also find that AI-generated solutions often follow coding style conventions well but score below human implementations on structural maintainability measures. Overall, the paper highlights the importance of evaluating both the final result and the reliability of the development process, while measuring maintainability primarily through tested functionality.

Critical strengths

A major strength of this paper is its focus on the reliability of the development process. Evaluating only the final code can hide repeated regressions and costly recovery cycles, which matter when agents work on shared, evolving software. SWE-CI makes these failures visible by tracking functional correctness across successive changes, while EvoScore distinguishes steady progress from unstable development even when agents reach the same final outcome. Grounding its 100 tasks in real development histories from 67 repositories also gives the benchmark practical relevance and broader coverage than evaluations built around isolated programming problems.

Another strength is the depth of its empirical analysis. Evaluating 31 models from nine providers allows the authors to identify patterns beyond individual model rankings, particularly the finding that stronger models distinguish themselves more consistently through lower regression frequency than smaller regression magnitude. The Architect–Programmer protocol also allows requirement analysis and implementation ability to be examined separately, helping identify where performance bottlenecks arise. Additional experiments test different agent frameworks, temporal weighting choices, and single-agent baselines, strengthening confidence in the main findings, although some checks cover only three models. Together, these analyses make the benchmark useful for understanding how agents fail and where improvements may help.

Critical weaknesses

The biggest weakness is that the paper treats passing-test counts as a proxy for maintainability. Tests show whether particular behaviors work, but they say much less about whether the code is understandable, well structured, or easy for another developer to change. Counting passing tests can also miss regressions: an agent could break one test and fix another without changing its score. EvoScore captures useful information about progress, but it mixes speed of improvement with stability, making it difficult to tell whether a high score reflects maintainable code or simply early gains on the target tests.

The setup also captures a narrower version of software maintenance than the paper’s framing suggests. Agents work toward a fixed test suite from a later commit, with requirements generated from the remaining failures. Real maintenance involves changing priorities, ambiguous requests, and dependency updates, whereas SWE-CI deliberately selects commit spans with unchanged dependencies. Its 100 tasks are also limited to Python repositories. Finally, the comparison with human code needs caution: humans may have made broader changes for reasons the benchmark does not capture, so differences in changed lines or static-analysis scores alone do not establish that agents avoid necessary refactoring.

The related-work discussion does not fully establish where SWE-CI sits among recent long-horizon software engineering benchmarks. SWE-Milestone is particularly relevant because it also examines continuous development, system integrity, and error accumulation. SWE Refactor Bench, SWE-Marathon, and DeepSWE cover other aspects of extended engineering work that deserve comparison, subject to the submission date. The paper already cites SWE-EVO and SWE-Bench-CL, but should explain more precisely what its evaluation captures beyond these benchmarks. A comparison of requirement progression, persistent repository state, regression tracking, and scoring would make the contribution much clearer.

Questions for authors
  1. How do you separate maintainability from simply making faster progress on the target tests? EvoScore rewards early gains, and passing-test counts can hide cases where one test breaks while another is fixed. Would tracking individual test regressions change the results or model rankings?

  2. How much does the Architect’s generated guidance influence performance? Since requirements come from failures against a fixed target test suite, could the benchmark favor agents that follow this feedback effectively over agents that handle ambiguous or changing maintenance requests?

  3. What does SWE-CI reveal that existing evolution benchmarks do not? A direct comparison with SWE-EVO, SWE-Bench-CL, and relevant concurrent work such as SWE-Milestone would help clarify whether the main contribution is the task construction, the CI workflow, or the trajectory-based scoring.

3 comments

Thank you for your review. Our responses are below.

W1 / Q1: Measuring maintainability with tests, regressions hidden by counting, and speed vs. stability in EvoScore

On using tests to measure maintainability. We agree that test results at a single point in time cannot show whether code is readable or well structured. However, SWE-CI does not measure the pass count at one point in time. It measures whether code stays correct under continuous modification, which is central to maintainability. ISO/IEC 25010 defines maintainability as the degree to which a system can be effectively and efficiently modified without introducing defects or degrading existing quality. Readability and good structure matter precisely because they make later changes less error-prone, and SWE-CI directly observes this consequence. For example, in Case 1 of Appendix D, the solution that only adds decoupled functions in each round has no regression at all, while the solution that makes large additions, deletions and refactorings in each round crashes three times. In contrast, static metrics do not reliably reflect maintainability, and different static metrics can even contradict each other (Wang et al., MaintainCoder, NeurIPS 2025).

We also tested whether EvoScore agrees with maintainability beyond tests. We selected 200 pairs in which two models both resolved the same task and one had a higher EvoScore. We told an LLM judge that both solutions are functionally correct, and asked it to compare their maintainability in terms of modularity, readability, extensibility and code style. The two judges (GLM-5.2 and Qwen3.7-Plus) preferred the higher-EvoScore solution in 137 (68.5%) and 131 (65.5%) pairs, and agreed with each other on 156 pairs. So EvoScore points in the same direction as broader maintainability, but the two are not the same. This is why the paper limits its scope to the testable aspect of maintainability (Abstract, §2.3). We will add this experiment to the appendix and narrow one remaining broad sentence in the introduction.

On regressions hidden by counting. This does happen. We re-ran the analysis by tracking individual tests: a round counts as a regression if any test that passed in the previous round fails in this round. Results (mean over 31 models):

MetricBy pass count (Table 2)By individual tests
Regression frequency17.09%28.19%
Zero-Reg28.77%16.54%

Among the rounds classified as "stable" or "progress" by pass count, 13.6% actually broke a previously passing test. However, the conclusion that stronger models regress less still holds. The Kendall τ between model rankings under the two definitions is 0.83 for regression frequency and 0.76 for Zero-Reg. Strong models (Res. Rate > 0.70, 7 models) have a mean regression frequency of 18.26%, versus 39.80% for weak models (Res. Rate < 0.30, 9 models). By pass count, these are 7.96% and 21.63%. The gap grows from 13.7 to 21.5 percentage points, and strong models still regress less than half as often. This also shows that keeping existing functionality intact under continuous modification is harder for current agents than our original numbers suggest. The revision will report regression frequency and Zero-Reg by individual tests.

On speed vs. stability. EvoScore includes both by design. To reach the same modification goal, maintainable code needs fewer iterations and less rework. The ISO/IEC 25010 definition of maintainability also covers both the effectiveness and the efficiency of modification. With γ = 1, EvoScore equals the area under the progress curve (for a resolved task, the remaining rounds count as a = 1; the revision will state this explicitly). It therefore rewards both avoiding regressions and reaching the goal sooner. To let readers separate the two, the paper reports regression frequency, regression magnitude, Zero-Reg and recovery rate separately (Table 2, Fig. 6). Moreover, rankings stay largely the same under different weights on early rounds: as γ goes from 0.8 to 1.2, Kendall τ is 0.85–0.95, and no model moves more than 5 positions (Table 2, Appendix C.1). A high EvoScore is therefore not driven by early gains alone.

W2: The setting is narrower than real-world maintenance

We agree that SWE-CI does not cover every form of software maintenance. However, it is closer to real maintenance than existing benchmarks:

  • Requirements are not fixed. In each round, the Architect generates new requirements from the current failures of the codebase. Regressions introduced by the agent itself become later requirements, so priorities change with the state of the codebase. Benchmarks that give all requirements at once do not have this property.
  • A fixed target version, as in all executable benchmarks, gives a verifiable end point. On average, each task spans 233 days and 71 commits, and involves 29.8 files and 1,592.7 changed lines (Table 1). This is far longer than a single issue.
  • Fixed dependencies keep the whole trajectory reproducible in one environment. Even under this constraint, tasks still span a long period (see above).
  • Python only, as in repository-level benchmarks such as SWE-bench, Commit0 and SWE-EVO. The evaluation paradigm itself is language-agnostic.

SWE-CI indeed does not include changing external requirements, ambiguous requirements, or dependency upgrades. The revision will add a Limitations section that states these limits.

On the comparison with human code. The observation in Table 3 holds: for all 31 models, human code has a higher Maintainability Index and more changed lines. But we agree that these differences do not show that agents avoid necessary refactoring. We will revise this statement accordingly.

W3 / Q3: How SWE-CI differs from existing long-horizon benchmarks

The paper already discusses SWE-EVO, SWE-Bench-CL and LongCLI-Bench (Appendix A). Along the four dimensions you proposed:

How requirements advancePersistent repository stateHow regressions are trackedScoring
SWE-EVOGiven once, from release notesNo, single taskOnly whether the final state keeps originally passing testsResolved Rate and Fix Rate of the final state
SWE-Bench-CLA chronological sequence of independent issuesNo; each issue starts from its own base commit, and only the agent's memory carries overNo code-regression tracking; focuses on forgetting across tasksResolution rate and continual-learning metrics
SWE-MilestoneA milestone sequence pre-built from commit historyYesPer test, at each milestonePrecision–recall score per milestone
SWE-CIRegenerated each round from current failuresYes, up to 20 roundsPer test, every round (see W1)Trajectory-level EvoScore, plus regression frequency, magnitude, Zero-Reg and recovery rate

SWE-Milestone (ICML 2026) is the closest work to ours. It was published shortly before the ICLR submission deadline. Both keep the repository state and both track regressions. They differ in three ways:

  • SWE-Milestone's requirements are fixed in advance. SWE-CI regenerates requirements each round from current failures, so the agent's earlier mistakes directly change later requirements.
  • SWE-CI records the full trajectory round by round. This allows us to analyze regression frequency, magnitude, trends over rounds, and recovery (§4 Q2–Q4).
  • The scale differs. SWE-CI has 100 tasks from 67 repositories and evaluates 31 models. SWE-Milestone has 98 milestones from 7 repositories and evaluates 12 models, but covers 5 languages.

SWE-Marathon, DeepSWE and SWE Refactor Bench were only available on arXiv at submission time, and they have different focuses. The first two evaluate a single very long task, and the third evaluates whole-repository migration. None of them evaluates regressions during continuous modification. The revision will cite and discuss them.

To answer Q3: the main contribution of this paper is trajectory-level evaluation and the regression patterns it reveals. The CI loop makes the codebase state observable at every round, and the task construction provides real start and end points.

Q2: The influence of the Architect, and whether the setup favors agents that are good at following feedback

The Architect matters, but less than the Programmer. In the 3×3 cross experiment with three GLM models (Table 5), changing the Architect with the Programmer fixed changes EvoScore by 0.073–0.105. Changing the Programmer with the Architect fixed changes it by 0.227–0.261. Whichever role is fixed, the order of the three models stays the same and matches their ranking in Table 2. So this setup can examine planning and implementation separately. The single-agent baseline is 0.028–0.051 lower than the dual-agent setup.

On whether the setup favors agents that are good at following feedback: making changes driven by test failures is exactly how real CI works. When a pipeline fails, developers locate and fix the problem from the failing tests, and this is the scenario SWE-CI evaluates. In addition, the Architect gives high-level requirements in natural language, at most five per round, and the Programmer still has to decide how to implement them. We agree that handling ambiguous or changing external requirements is a different and important capability. It is outside the scope of SWE-CI, and the revision will state this in the Limitations section.

We hope these responses address your concerns. If so, we would be grateful if you would consider raising your score.

Comments on the paper

1 comment

Dear AC and reviewers, We thank all reviewers for their comments. We have uploaded a revised version of the paper. All changes are marked in blue, the main text remains within the page limit, and the rest of the paper is unchanged.

  • Related work (Reviewers gWVz and X1Za). Appendix A now discusses SWE-Milestone, MaintainCoder, SWE-Marathon, DeepSWE and SWE Refactor Bench.
  • Regressions tracked per test (Reviewers X1Za and TyJY). A new Appendix F.1 (Table 11) reports regression frequency, Zero-Reg and recovery rate by individual tests. Q2 in Section 4 now points to it.
  • Comparison with judged maintainability (Reviewers X1Za and TyJY). Appendix F.2 reports the pairwise comparison with two LLM judges.
  • Run-to-run variation (Reviewer TyJY). Appendix F.3 reports two additional runs of GLM-5. The caption of Table 2 now notes that models within about 0.03 EvoScore form one tier.
  • Scope (Reviewers X1Za and gWVz). A new Appendix G states the limits of the setting. The introduction now refers to "a testable aspect of maintainability". Q5 in Section 4 now describes the comparison with human code without inferring a cause.
  • Dual-agent protocol (Reviewer TyJY). Q7 in Section 4 now names the two capabilities it separates: requirement analysis based on CI feedback, and implementation.
  • EvoScore definition (Reviewer X1Za). Section 2.3 now states that the remaining iterations of a resolved task take a = 1.
  • Minor corrections. Q1 and Q2 in Section 4 and Appendix C.1 now use statistics computed over all 31 models in Table 2. Section 2.3 quotes the ISO/IEC 25010 definition in full. The numbering of Appendices A–E and Tables 1–10 is unchanged, so the references in our earlier responses still apply.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.