- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SWE-CI, a benchmark designed to evaluate the ability of coding agents to continuously maintain codebases over the long term. The benchmark constructs 100 tasks based on real GitHub repository histories, utilizing an Architect–Programmer dual-agent CI loop to iteratively modify and correct code. Additionally, it proposes an EvoScore to track correctness changes throughout the development process. Finally, the paper evaluates a broad range of model families and finds that they often suffer from severe functional regressions during development.
Critical strengths
- Valuable evaluation setting and comprehensive experiments. This paper proposes analyzing the entire trajectory across the modification history of the codebase, and provides an in-depth discussion on intermediate regressions that the final success rate cannot capture. The comprehensive experiments across a wide range of model families are also meaningful for drawing general conclusions and would benefit the SWE community.
Critical weaknesses
-
Limited novelty due to missing discussion with closely related work. MaintainCoder [1] and SWE-Milestone [2] have already focused on evaluating agents' abilities to maintain dynamic codebases, with a particular focus on regression and evolution during this process. This paper does not clearly discuss these works, which weakens its novelty and contribution.
-
The design remains oversimplified compared to real-world code maintenance. When resolving long-range commit histories, the agent only has access to the start and target versions without replaying real-world step-by-step development. Furthermore, the agent has access to the target version's test cases, making it more of an iterative development setting rather than long-term codebase maintenance. Additionally, the paper requires that dependencies cannot change between the start and target versions, which also diverges from long-term real-world SWE scenarios.
[1] Wang, Zhengren, et al. "Maintaincoder: Maintainable code generation under dynamic requirements." Advances in Neural Information Processing Systems 38 (2026): 15399-15436.
[2] Deng, Gangda, et al. "SWE-Milestone: Evaluating AI agents on continuous software evolution." International Conference on Machine Learning. PMLR, 2026.
Questions for authors
-
What are the core differences between the proposed benchmark and existing work such as MaintainCoder and SWE-Milestone? Could the authors provide a more systematic comparison?
-
SWE-CI evaluates code evolution toward a fixed target version and excludes commit spans with dependency changes. How do the authors justify the benchmark's representativeness in reflecting realistic, long-term software evolution?
3 comments
2/2
W2 / Q2: Simplifications of the setting, and representativeness
We agree that SWE-CI does not cover every form of long-term evolution. It represents the most common one: continuously extending and modifying an existing codebase in a stable environment. We believe the following simplifications do not affect its representativeness for this setting:
- Not replaying intermediate historical steps. This is a deliberate design choice. In real maintenance, what to do next is decided by the current state of the codebase, not by a pre-written path. In each round, SWE-CI generates at most five requirements from the current failures, and across 31 models each task takes 13.9 CI iterations on average (Table 2). Replaying history step by step evaluates the ability to follow a known path. SWE-CI evaluates the ability to keep existing functionality intact while making progress on one's own. The two are complementary.
- Access to the target version's tests. In CI, the test suite is part of the codebase. When a pipeline fails, developers read the failing tests to locate the problem. Under ISO/IEC/IEEE 14764, modifications made after delivery to add or improve functionality are themselves software maintenance (the enhancement category). Moreover, seeing the tests does not make the task easy. Tracking individual tests, 28.19% of rounds break a previously passing test on average across 31 models, and only 16.54% of tasks are free of regressions throughout. The difficulty is not knowing the goal, but avoiding breaking existing functionality under continuous modification.
- A fixed target version. This gives a verifiable end point. Every executable repository-level benchmark needs such an end point; in SWE-Milestone, each milestone is also verified against a historical version.
- Fixed dependencies. This keeps the whole trajectory in one environment, so test failures can only come from the agent's code, not from changes in the environment. Among all maximal commit spans with unchanged dependencies, we selected the 100 with the longest time span and the most commits. On average, these tasks span 233 days and 71 commits, and involve 29.8 files and 1,592.7 changed lines (Table 1).
Dependency upgrades and changing external requirements are outside the scope of SWE-CI. The revision will state this in a new Limitations section.
We hope these responses answer your questions. We are happy to discuss further if anything remains unclear.
Thank you to the authors for the detailed responses and additional experiments. I will maintain my positive score.
1/2
Thank you for your review, and for recognizing the value of trajectory-level evaluation and the scale of our experiments. Our responses are below.
W1 / Q1: Differences from MaintainCoder and SWE-Milestone
Thank you for pointing out these two related works. The revision will discuss and cite them in the related work section. Here is a comparison of the three:
The core differences: