acceptodds
Under review as a conference paper at ICLR 2027

SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

Abstract

Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads to an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce **SWE Refactor Bench**, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs (5.4%) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores 47.0/100. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, 58% reach 99% of the fixed checks, yet only 26% reach 100%. Agent capability differs across migration categories: agents score 31.4 on build toolchain rewrites but only 5.6 on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.

Primary area:
datasets and benchmarks
Keywords:
coding agents, benchmark, code migration, software engineering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gBDrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper benchmarks coding agents on whole-repository stack migrations where behaviour must be preserved while the stack changes (C to Rust, Autotools to CMake, POSIX to wasm32-wasi, etc.). The authors formalise a failure mode they call Blindness. Section 2.1 argues that for behaviour-preserving tasks the starting repository already passes its own tests, so returning the repository unchanged earns full marks on any behavioural suite by construction, and enlarging the suite cannot close this gap. SWE Refactor Bench contains 20 real OSS migrations across four debt classes (language, framework, platform, build toolchain), 130,118 fixed checks across 264 modules, and 136 Migration-Audit criteria. The three-stage protocol applies in order: Migration Audit by an LLM judge (gpt-5.6-sol) with majority-of-three independent samples per criterion and a hard veto, 130,118 fixed Behavioural Tests all-or-nothing, and six independent agentic verifiers (one hour each) that may reject only by producing an executable counterexample that passes on the original and fails three times on the submission. Across 520 runs from 8 frontier models in 26 effort configurations, 28 of 520 (5.4%) pass all three stages, 13 of 20 tasks are never solved, and claude-opus-5 at xhigh effort leads at 47.0/100 against 28.5 for gpt-5.6-sol and 19.5 for kimi-k3. The paper documents two distinct failure modes with roughly 8:1 ratio (252 completed the migration but broke behaviour, 30 preserved behaviour by skipping the migration), the "last 1%" phenomenon (58% of Stage I passers reach 99% of fixed checks but only 26% reach 100%), and the Stage III take-back (60 of 88 Stage II passers are broken by at least one verifier). Appendix E reports 89.7% / Cohen kappa 0.795 human agreement on 156 annotated runs across 6 tasks, no self-family collusion in judge or verifiers under a proper control, and non-redundant exclusive-break contributions across all six verifiers.

Critical strengths

1. Blindness is a principled contribution and the three-stage design operationalises it with no stage made redundant. Section 2.1 is a formal argument about the behaviour-preservation task family rather than a rhetorical frame: because the untouched repository passes every observation drawn from the original by construction (rate(R_A; T) = 1 for any T ⊆ O), no behavioural suite can distinguish an unmigrated repository from a correctly migrated one. The three-stage protocol follows directly from this: Stage I is a non-behavioural veto, Stage II checks a drop-in-replacement property that is all-or-nothing on purpose, and Stage III probes residues no fixed suite was written to catch. The empirical funnel in Table 1 confirms the design is load-bearing: 30 runs are blindness (passed Stage II but vetoed at Stage I), 252 are the opposite (passed Stage I but failed Stage II), and 60 more pass both Stage I and Stage II yet are broken at Stage III. The three stages are measurably disjoint, which is the right evidence that each is necessary. The "last 1%" analysis (Figure 3a) and the specific regression examples (fw03 hash routing, build03 blank PyPI METADATA) show the fixed-suite stage is not just a formality: at 99.95% pass rate, 18 submissions still miss exactly one check and each miss corresponds to a visible production regression.

2. Validity analysis in Appendix E is unusually thorough for a benchmark paper and anticipates the obvious reviewer objections. Appendix E reports Stage I inter-sample agreement (96.3% unanimous across three independent samples, and 305 vs 340 would pass without majority voting, so majority voting is load-bearing not cosmetic), independent human annotation of 156 runs with Cohen kappa 0.795 and error direction skewed toward being stricter than lenient (14 of 16 disagreements are the judge too strict, and following all 16 cases through changes at most 2 final verdicts), and a direct test for same-family collusion (Finding 2) that rules out collusion by a within-model control: opus verifiers break 40.8% of opus submissions vs 65.0% of other-family submissions, but the four non-opus verifiers on the same submissions show the same 17.1 vs 29.1% ratio, so the gap is submission quality rather than family-preference. The verifier panel is also shown to be non-redundant: counting exclusive breaks, even the lowest-break-rate verifier rejected a submission that the other five let through.

Critical weaknesses

1. The Stage III leaderboard is panel-specific and the paper does not precommit to a protocol for refreshing the verifier panel over time. Finding 1 in Appendix E openly states that retiring the two claude-opus-5 verifiers (break rates 55.7% and 53.4%, versus 21.6%-26.1% for the other four) would move accepted submissions from 28 to 46, which is a near-doubling of the top-end rank. Section 2.1 argues the number of surviving verifiers is "evidence, not a demonstration of equivalence", and Finding 1 concludes "as models improve, this same fixed set of 20 tasks will be scored more strictly". These two statements together say the headline leaderboard numbers are not cross-year comparable without a defined refresh protocol: if the panel is held fixed, the benchmark becomes easier over time; if the panel is refreshed when stronger models appear, numbers from different years cannot be compared at all. For a benchmark paper that explicitly positions itself as a long-term testbed, this is a significant unresolved design question, not a minor caveat.

2. Twenty tasks, one run per model-configuration-task cell, and no cell-level uncertainty is reported on the leaderboard. Each row of Table 1 is 20 independent Bernoulli-like outcomes with acceptance rates in the 5 to 25 percent range, and no model-configuration cell is repeated with different seeds or temperature. The per-model best-configuration ranks (claude-opus-5 at 5 of 20 accepted and 47.0 score, gpt-5.6-sol at 3 of 20 and 28.5, kimi-k3 at 2 of 20 and 19.5) are therefore small-integer differences without reported confidence intervals, and because both the submission agents and the six Stage III verifiers are nondeterministic hour-scale agentic systems, the resampling variance is not pinned down. Stage I uses majority of three samples and Stage III requires three successful reproductions, so variance reduction is built in at the stage level, but not at the model-level comparison level. For a benchmark that will be used to pick between models for years, at least one subset with three or more seeds per cell is a reasonable minimum.

Questions for authors
  1. Panel-refresh protocol. Do you have a planned protocol for refreshing the verifier panel as stronger coding agents become available, and if so, how will acceptance counts be reported such that numbers from different panel generations remain interpretable? If no refresh is planned, do you have evidence that the panel will remain discriminating for a reasonable period?

  2. Seed-level variability on a subset. Can you report three or more seeds per cell for at least the seven framework-rewrite tasks (which contribute 14 of 28 acceptances), so the gap between the leader cluster (47.0, 28.5) and the next (19.5, 15.0) can be read with confidence intervals rather than as point estimates? Even a single-model, single-task bootstrap would help calibrate what count-of-one differences mean in Table 1.

  3. Scoring-weight robustness. Equation (4) weights Stage III at 0.6 with a linear term 0.4 + 0.6 s/6 in surviving verifiers. How does the model ranking change under (a) Stage III as all-or-nothing (s = 6 required), (b) 0 + s/6 instead of 0.4 + 0.6 s/6, and (c) Stage III weight 0.3 instead of 0.6? Rank robustness here would strengthen the leaderboard interpretation beyond the specific chosen split.

  4. Blindness per task. The 30 Blindness runs are reported in aggregate and the paper notes that on lang01 five of six Stage I passers were Blindness, and on fw01, fw02, fw07 every passing submission was Blindness. Can you provide the per-task count for all 20 tasks? This is needed to assess the diagnostic value of Stage I separately from Stage II, and to understand which task designs most invite Blindness (presumably those where a word-for-word transliteration of the source implementation is permitted by the target language).

  5. Harness confound. The two GPT-series models run under the Codex harness while the other six run under Claude Code. Have you tested at least one non-GPT model under Codex, or a GPT model under Claude Code, to confirm that the measured cross-family gaps are not driven by the harness choice?

Reviewer_2U2wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWE Refactor Bench asks agents to migrate 20 real infrastructure repositories to a new stack, scoring each submission with a three-stage protocol (migration audit, fixed behavioral tests, and differential verification by six agents) that checks whether the migration happened and whether behavior is preserved. Only 5.4% of 520 runs across 8 models pass all three stages, and the best model scores 47.0/100.

Critical strengths
  1. Behavioral tests alone can't tell a completed migration from a repository handed back unchanged, and the authors identify this problem precisely. The three-stage protocol targets exactly this gap, and Stage III accepts only executable counterexamples.
  2. The tasks are real and substantial. Appendix E checks the judge's agreement with human annotators and tests for same-family favoritism, and logs for all 520 runs are released.
Critical weaknesses
  1. Eq. (2) only requires the old stack to be gone from the repository and the build, yet the 5 Blindness cases on lang01 were zeroed for transliterating the original into unsafe Rust. Is the Stage I veto then consistent with the paper's own definition of migration?
  2. Appendix E.3 acknowledges that removing the two Opus verifiers would raise acceptances from 28 to 46, and each configuration runs each task only once. What evidence shows that the leaderboard reflects model capability, not the makeup of the verifier panel or single-run randomness?
Questions for authors
  1. Does instruction I explicitly require a rewrite that isn't a transliteration? If not, how should the Blindness counts on lang01 and similar tasks be interpreted?
  2. Tasks ship without reference solutions, and the identity run only validates State A against itself. For the 13 tasks no model solved, how do you show they are solvable within the given budget?
Reviewer_ESroOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SWE Refactor Bench, a benchmark of 20 whole-repository stack migrations spanning implementation languages, application frameworks, platforms, and build toolchains. Its central motivation is that behavioural tests alone cannot establish whether a requested migration has actually occurred, because the unchanged original implementation may already pass those tests.

The proposed evaluation combines three stages: a rubric-based migration audit, fixed behavioural tests, and six coding agents that generate executable differential counterexamples after submission. The paper evaluates eight models across 26 model–effort configurations, for 520 runs, of which 28 pass all three stages. Among the 88 submissions that pass the first two stages, 60 are subsequently broken by the agentic verifiers. Additional analyses examine agreement between migration audits and human annotations, as well as the contributions of individual verifiers.

Critical strengths

1. A practically important benchmark setting

Whole-repository stack migration is an important software-maintenance problem requiring coordinated changes across implementation, dependencies, interfaces, and build systems. The benchmark covers language, framework, platform, and build-toolchain migrations, with preservation of existing behaviour as a constraint rather than evaluating only localized issue resolution. Its 20 tasks and 520 runs provide a useful empirical testbed. I see the main contribution as the task setting and systematic evaluation; a benchmark contribution need not also introduce a new migration algorithm or address every form of architectural technical debt.

2. Verification grounded in executable counterexamples

The paper does not equate passing fixed tests with successful migration. It first audits stack replacement and then requires verification agents to produce runnable, reproducible differential counterexamples. Finding counterexamples for 60 of the 88 submissions that pass the first two stages demonstrates that dynamic verification supplements the fixed suite's coverage. Human audit comparisons and verifier ablations also provide useful validation. The composite score and strict acceptance rate should be interpreted separately, but stage-dependent scoring is not, by itself, a decisive objection to the work.

Critical weaknesses

1. The alignment between fixed tests and the migration contract needs clearer justification

Using the original system as a behavioural reference is a reasonable approach to compatibility testing. However, cross-language and cross-framework migration requires distinguishing public interface behaviours and boundary cases relied upon by downstream users from implementation details or behaviours outside the agreed scope. An output produced by the original system does not, by itself, establish that every detail of that output should be a mandatory migration requirement.

The paper already uses reference self-testing, author review, and allow/deny scopes for the third stage to reduce evaluation errors. Nevertheless, because any fixed-test failure yields zero credit, the criteria for including and excluding behaviours are consequential. A clearer explanation of how tests map to the task contract, particularly for source-language- or source-framework-specific behaviours, would strengthen the evaluation. This is not a request to relax genuine compatibility requirements, nor a claim that the existing tests are invalid; it is a request for a more explicit justification of test selection.

The same clarification applies to the migration audit. The paper's formal conditions emphasize stack replacement and behavioural preservation, whereas the transliteration example in Table 3 also invokes ownership redesign and the safety benefits expected from the migration. The manuscript should clarify whether the audit measures stack replacement alone or additionally requires a particular design, safety property, or implementation quality. Such additional objectives can be legitimate, but should be explicit in the agent-visible requirements and assessed using clearly stated criteria rather than implicitly equated with whether migration occurred. I view this as a task-boundary clarification rather than grounds for dismissing the evaluation based on that example alone.

2. The engineering motivation for some framework migrations needs more concrete explanation

The paper describes selecting technical debt before selecting repositories. For representative framework migrations, however, it would be helpful to explain more concretely what maintenance or operational limitation the original framework creates, why the target framework is appropriate, and what form of technical debt the migration is expected to remove.

Replacing a framework while preserving existing behaviour can have practical value without introducing new functionality. However, when a task requires closely reproducing the original framework's default behaviours, readers need to understand which behaviours are necessary compatibility commitments, which are incidental characteristics, and how compatibility costs relate to the intended benefits.

This clarification matters to the benchmark's practical representativeness. It does not require proving that every migration improves performance or maintainability, or extending the benchmark to architectural refactoring. I consider the current task setting valuable, but a more concrete account of the engineering motivation would make the contribution more persuasive.

Questions for authors
  1. Test-contract clarification: How do the fixed behavioural tests map to the agent-visible migration requirements? What criteria distinguish required external compatibility from internal or incidental source-language/framework behaviours? Please provide examples of included and excluded behaviours and the rationale for those decisions.

  2. Framework-migration motivation: For one or two representative framework tasks, could you explain the concrete limitations of the original stack, the rationale for choosing the target framework, and the associated maintenance need or expected engineering benefit? How was the scope of exact compatibility with the original framework's default behaviours determined?

  3. Stack replacement versus implementation-quality objectives: Does Migration Audit assess whether the old stack has been replaced, or also whether the submission follows a preferred target-language design? In Table 3 and the cmark discussion, a submission is rejected for transliteration without ownership redesign despite passing the behavioural suite. Which agent-visible requirements mandate that redesign or the associated safety benefit? Please distinguish stack replacement from any additional safety or implementation-quality objectives, and explain how those objectives are assessed. I am not arguing that every transliteration should be accepted, but that the acceptance contract should make this distinction explicit.

  4. Controlled cross-harness evaluation: Section 3.1 uses Codex for the two GPT models and Claude Code for the remaining six, so model-level comparisons also vary the harness. Could you evaluate a representative subset with the same models across multiple harnesses, including a lightweight alternative such as mini-SWE-agent? Holding task inputs, tool access, model effort, and comparable execution budgets fixed would help separate model capability from harness effects. Please report task acceptance, stage-wise failures, and resource use, and indicate whether the relative model ordering is stable. If such a comparison is not feasible, please explicitly scope the results as model–harness configurations rather than isolated model capabilities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.