- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper benchmarks coding agents on whole-repository stack migrations where behaviour must be preserved while the stack changes (C to Rust, Autotools to CMake, POSIX to wasm32-wasi, etc.). The authors formalise a failure mode they call Blindness. Section 2.1 argues that for behaviour-preserving tasks the starting repository already passes its own tests, so returning the repository unchanged earns full marks on any behavioural suite by construction, and enlarging the suite cannot close this gap. SWE Refactor Bench contains 20 real OSS migrations across four debt classes (language, framework, platform, build toolchain), 130,118 fixed checks across 264 modules, and 136 Migration-Audit criteria. The three-stage protocol applies in order: Migration Audit by an LLM judge (gpt-5.6-sol) with majority-of-three independent samples per criterion and a hard veto, 130,118 fixed Behavioural Tests all-or-nothing, and six independent agentic verifiers (one hour each) that may reject only by producing an executable counterexample that passes on the original and fails three times on the submission. Across 520 runs from 8 frontier models in 26 effort configurations, 28 of 520 (5.4%) pass all three stages, 13 of 20 tasks are never solved, and claude-opus-5 at xhigh effort leads at 47.0/100 against 28.5 for gpt-5.6-sol and 19.5 for kimi-k3. The paper documents two distinct failure modes with roughly 8:1 ratio (252 completed the migration but broke behaviour, 30 preserved behaviour by skipping the migration), the "last 1%" phenomenon (58% of Stage I passers reach 99% of fixed checks but only 26% reach 100%), and the Stage III take-back (60 of 88 Stage II passers are broken by at least one verifier). Appendix E reports 89.7% / Cohen kappa 0.795 human agreement on 156 annotated runs across 6 tasks, no self-family collusion in judge or verifiers under a proper control, and non-redundant exclusive-break contributions across all six verifiers.
Critical strengths
1. Blindness is a principled contribution and the three-stage design operationalises it with no stage made redundant. Section 2.1 is a formal argument about the behaviour-preservation task family rather than a rhetorical frame: because the untouched repository passes every observation drawn from the original by construction (rate(R_A; T) = 1 for any T ⊆ O), no behavioural suite can distinguish an unmigrated repository from a correctly migrated one. The three-stage protocol follows directly from this: Stage I is a non-behavioural veto, Stage II checks a drop-in-replacement property that is all-or-nothing on purpose, and Stage III probes residues no fixed suite was written to catch. The empirical funnel in Table 1 confirms the design is load-bearing: 30 runs are blindness (passed Stage II but vetoed at Stage I), 252 are the opposite (passed Stage I but failed Stage II), and 60 more pass both Stage I and Stage II yet are broken at Stage III. The three stages are measurably disjoint, which is the right evidence that each is necessary. The "last 1%" analysis (Figure 3a) and the specific regression examples (fw03 hash routing, build03 blank PyPI METADATA) show the fixed-suite stage is not just a formality: at 99.95% pass rate, 18 submissions still miss exactly one check and each miss corresponds to a visible production regression.
2. Validity analysis in Appendix E is unusually thorough for a benchmark paper and anticipates the obvious reviewer objections. Appendix E reports Stage I inter-sample agreement (96.3% unanimous across three independent samples, and 305 vs 340 would pass without majority voting, so majority voting is load-bearing not cosmetic), independent human annotation of 156 runs with Cohen kappa 0.795 and error direction skewed toward being stricter than lenient (14 of 16 disagreements are the judge too strict, and following all 16 cases through changes at most 2 final verdicts), and a direct test for same-family collusion (Finding 2) that rules out collusion by a within-model control: opus verifiers break 40.8% of opus submissions vs 65.0% of other-family submissions, but the four non-opus verifiers on the same submissions show the same 17.1 vs 29.1% ratio, so the gap is submission quality rather than family-preference. The verifier panel is also shown to be non-redundant: counting exclusive breaks, even the lowest-break-rate verifier rejected a submission that the other five let through.
Critical weaknesses
1. The Stage III leaderboard is panel-specific and the paper does not precommit to a protocol for refreshing the verifier panel over time. Finding 1 in Appendix E openly states that retiring the two claude-opus-5 verifiers (break rates 55.7% and 53.4%, versus 21.6%-26.1% for the other four) would move accepted submissions from 28 to 46, which is a near-doubling of the top-end rank. Section 2.1 argues the number of surviving verifiers is "evidence, not a demonstration of equivalence", and Finding 1 concludes "as models improve, this same fixed set of 20 tasks will be scored more strictly". These two statements together say the headline leaderboard numbers are not cross-year comparable without a defined refresh protocol: if the panel is held fixed, the benchmark becomes easier over time; if the panel is refreshed when stronger models appear, numbers from different years cannot be compared at all. For a benchmark paper that explicitly positions itself as a long-term testbed, this is a significant unresolved design question, not a minor caveat.
2. Twenty tasks, one run per model-configuration-task cell, and no cell-level uncertainty is reported on the leaderboard. Each row of Table 1 is 20 independent Bernoulli-like outcomes with acceptance rates in the 5 to 25 percent range, and no model-configuration cell is repeated with different seeds or temperature. The per-model best-configuration ranks (claude-opus-5 at 5 of 20 accepted and 47.0 score, gpt-5.6-sol at 3 of 20 and 28.5, kimi-k3 at 2 of 20 and 19.5) are therefore small-integer differences without reported confidence intervals, and because both the submission agents and the six Stage III verifiers are nondeterministic hour-scale agentic systems, the resampling variance is not pinned down. Stage I uses majority of three samples and Stage III requires three successful reproductions, so variance reduction is built in at the stage level, but not at the model-level comparison level. For a benchmark that will be used to pick between models for years, at least one subset with three or more seeds per cell is a reasonable minimum.
Questions for authors
-
Panel-refresh protocol. Do you have a planned protocol for refreshing the verifier panel as stronger coding agents become available, and if so, how will acceptance counts be reported such that numbers from different panel generations remain interpretable? If no refresh is planned, do you have evidence that the panel will remain discriminating for a reasonable period?
-
Seed-level variability on a subset. Can you report three or more seeds per cell for at least the seven framework-rewrite tasks (which contribute 14 of 28 acceptances), so the gap between the leader cluster (47.0, 28.5) and the next (19.5, 15.0) can be read with confidence intervals rather than as point estimates? Even a single-model, single-task bootstrap would help calibrate what count-of-one differences mean in Table 1.
-
Scoring-weight robustness. Equation (4) weights Stage III at 0.6 with a linear term 0.4 + 0.6 s/6 in surviving verifiers. How does the model ranking change under (a) Stage III as all-or-nothing (s = 6 required), (b) 0 + s/6 instead of 0.4 + 0.6 s/6, and (c) Stage III weight 0.3 instead of 0.6? Rank robustness here would strengthen the leaderboard interpretation beyond the specific chosen split.
-
Blindness per task. The 30 Blindness runs are reported in aggregate and the paper notes that on lang01 five of six Stage I passers were Blindness, and on fw01, fw02, fw07 every passing submission was Blindness. Can you provide the per-task count for all 20 tasks? This is needed to assess the diagnostic value of Stage I separately from Stage II, and to understand which task designs most invite Blindness (presumably those where a word-for-word transliteration of the source implementation is permitted by the target language).
-
Harness confound. The two GPT-series models run under the Codex harness while the other six run under Claude Code. Have you tested at least one non-GPT model under Codex, or a GPT model under Claude Code, to confirm that the measured cross-family gaps are not driven by the harness choice?