acceptodds
Under review as a conference paper at ICLR 2027

SWE-Duel: Towards A Self-Scaling Arena for Coding Agents

Abstract

Static coding benchmarks are fixed at release. Over time their tasks leak into training data, and stronger agents eventually saturate them. We argue that a benchmark should instead aim to be self-scaling: its tasks and oracles are created at evaluation time, so no content exists in advance to leak, and its difficulty is set by the participating agents rather than fixed at release. We introduce SWE-DUEL, an arena in which two coding agents evaluate each other over mirrored Red–Blue turns. In each turn, the Red agent works on a real repository across three context-isolated sessions. It proposes and implements a new feature with feature tests, then embeds a subtle bug in that feature along with sealed bug-triggering tests. The resulting pull request is admitted only if (1) it passes the feature tests and the repository’s existing tests while failing exactly the bug tests, and (2) a fresh session of the same agent can repair the bug blind, which rules out unsolvable tasks. The Blue agent then reviews the pull request without being told a bug exists. It wins only if its revision passes all three test suites, and then the two agents swap roles. We run a tournament among 11 model–harness participants on 15 repositories in five languages for 1,500 turns. The leaders win in different ways. GPT-5.5 with Codex CLI ranks first as the strongest solver, repairing 98.3% of the defects it reviews. Claude Opus 4.8 with mini-swe-agent ranks second as the strongest task author, with a 27.3% Red win rate. Finally, within one model family, tasks written by the newest version defeat a fixed solver far more often than those of older versions (42.9% vs. 4.0%), giving preliminary evidence that task difficulty scales with task authoring capability.

Primary area:
datasets and benchmarks
Keywords:
Coding agents, Benchmark, Arena

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_7jqDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWE-Duel evaluates coding agents through reciprocal task generation and repair, using feature-preservation and hidden bug tests. The experiments cover 11 model–harness pairs and 15 repositories.

Critical strengths

Executable tests provide concrete evaluation criteria. The fixed-solver experiments and same-model harness comparisons also provide useful controls.

Critical weaknesses
  1. The current tasks are close to saturation for the strongest solvers. GPT-5.5 achieves repair success rates of 98.3% and 96.4% on contested tasks (Table 8). Table 2 shows that stronger task authors can challenge a fixed solver, but does not establish that task difficulty keeps pace as frontier solvers improve. Appendix L acknowledges this bottleneck. I find a substantial gap between the motivation of overcoming benchmark saturation and the evidence presented.

  2. The self-repair admission rule may restrict task difficulty and coverage. The bank retains only tasks on which the author has already succeeded at blind self-repair. Valid tasks that expose the author’s own weaknesses can therefore be discarded. Swapping roles does not remove this selection effect. The paper does not sufficiently establish which useful hard tasks this gate excludes or how that exclusion affects the capabilities being evaluated.

Questions for authors
  1. Which results support continued discrimination among improving frontier solvers?
  2. How many otherwise valid tasks does the self-repair gate reject, and what do those tasks reveal about lost difficulty or capability coverage?
Reviewer_3PEkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark where coding agents hide bugs in new features and review each other's changes. Across 11 model-harness pairs, GPT-5.5/Codex repairs 98.3% of contested tasks. A study also suggests stronger authors can create harder tasks.

Critical strengths

The setup is useful for testing code review. The visible tests pass, so agents have to find bugs without an obvious failing test that points them out. Feature and regression tests check that a repair keeps the feature and existing behavior.

Critical weaknesses
  1. The evidence for continued self-scaling is limited. The top solver already repairs 98.3% of the tasks. The GLM study shows harder tasks for a fixed solver, but it doesn't show that the new tasks remain challenging as both authors and solvers improve. Gate C not only has the potential problem of actually filtering out difficult tasks that would distinguish the top solves, but by itself also cannot guarantee that a task has no errors in its hidden tests etc.
  2. The exact ranking among the top agents remain uncertain. Each participant only contributes 5-15 tasks, which are reused against several opponents. Bootstrapping matches may not capture how much of the ranking depends on which tasks were generated. Appendix N resamples tasks but also switches to turn-level scoring, so it doesn't directly check uncertainty in the original ranking. The paper also reports that no adjacent ranks are distinguishable at 95% (Appendix H). The authors should resample tasks while keeping the original match scoring. They should also report whether correct the hidden-test errors (in Appendix G) changes the match outcomes or rankings.
Questions for authors
  1. Do the harder tasks from new authors remain challenging for the strongest tournament solvers? Could you also inspect a sample of tasks rejected by Gate C to check whether it filters out valid tasks that would distinguish the top solvers?
  2. How do the rankings and CIs change if you resample tasks, recompute match outcomes, and refit the original match-level model?
  3. Did you correct for the hidden-test error found? Are all affected submissions rescored? If so, did any match outcomes or rankings change? If not, could you report this check?
Reviewer_vCPVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Static benchmarks eventually leak into model training data, and stronger agents saturate on them. This paper argues that a benchmark should be self-scaling: the corresponding tasks and oracles should be created at test time to avoid leakage. To achieve this, the authors introduce SWE-Duel, an arena in which it sets two coding agents against each other. The red agent works on a real repository to induce a new feature as PR along with subtle bug and sealed tests to test for this bug. The blue agent reviews the PR without being told about the bug. It wins, if it identifies the bug and passes all tests. The roles are then swapped. Extensive experiments iterating over 11 model-harness participants demonstrates preliminary evidence of self-scaling: difficulty should rise as those agents improve.

Critical strengths

I found two strengths noteworthy. One, SWE-Duel is a simple and effective design to avoid test set leakage, a possible issue for static benchmarks. Second, several existing benchmarks only check code generation and bug localization abilities. The red agent’s task to author a new feature and introducing a subtle bug also captures a new axis.

Critical weaknesses

My main concern with the paper is that it argues that task difficulty scales with task authoring capability. However, the experiment setup admits a task only if author (red agent) can solve the induced subtle bug, without knowing about it. This constraint, by design, limits the task difficulty to not exceed the agent’s own solving ability. The tournament (Table 1) also looks saturated, weakening the self-scaling claim.

Questions for authors

Can the authors validate if the red agent wins are not because of underspecified spec and subtle bugs indeed exist? Also, Appendix A mentions that task bank should be regenerated for every tournament. Does ranking hold with a new task bank?

Reviewer_QXeNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article argues that once a static coding benchmark is released, it will leak and eventually become saturated. It advocates that the benchmark should be "self scaling": tasks and oracle are only generated during evaluation, and the difficulty is determined by the participants themselves. The author proposes SWE-DUEL based on this: two encoding agents perform a mirrored Red Blue game on a real repository. Red uses three isolated sessions to first implement new features and write functional tests, and then embeds hidden bugs in the features and writes closed bug tests; Tasks are only included if they pass functional testing and existing testing, happen to fail bug testing, and Red's own new session can blindly fix the bug. Blue reviewed the PR without knowing there were bugs, and only by passing three sets of tests at the same time after the revisions were made, would it be considered a win. Subsequently, both sides exchanged roles. The author conducted a tournament on 15 repositories (5 languages) with 11 model harness combinations for 1500 turns: GPT-5.5+Codex CLI ranked first (with a repair rate of 98.3%), Claude Opus 4.8+mini swe agent ranked second and had the strongest problem-solving ability (Red win rate of 27.3%); New participants only need about 5 matches to enter the list; In the same model family (GLM-5.1/5.2/5.3), the proportion of fixed solver failures caused by the latest version of questions is higher (42.9% vs. 4.0%), and the author believes that task difficulty increases with the ability to produce questions.

Critical strengths
  • Integrating the "task generation during evaluation" and "solvability verification" into the same protocol, embedding bugs within the new feature makes it impossible for Blue to win by rolling back, and the blind self repair threshold eliminates unsolvable tasks. The design concept is clear and innovative.
  • The author has conducted a lot of supporting analysis (fixed reference solver, admission stability, manual audit of bug testing misjudgment, automatic victory ablation, harness ablation, cost), and is also quite frank about the limitations of self repair upper limits and overlapping confidence intervals. New participants only need a small number of competitions and about $243 to access, which is highly practical.
Critical weaknesses
  • The core proposition of the paper is that task difficulty increases with problem-solving ability, but the tournament itself cannot test this point. Direct evidence only comes from small-scale comparisons of three versions of the same GLM family, with few tasks in each version and only one external solver used. It is difficult to judge whether the conclusion is universal.
  • The number of matches and tasks between participants is not large, and adjacent rankings are basically indistinguishable. After removing the automatic victory judgment of missing tasks, the ranking will still change significantly. Therefore, the specific ranking is more like the result of design selection rather than a stable conclusion.
  • The specifications and closed bug tests were both written by Red itself, and "solvable" only indicates that its own new script can guess the same intention. The manual audit only sampled a portion of the failed cases, and how much of Red's victory came from the lack of independent verification of specification ambiguity.
Questions for authors
  • There have been significant changes in the rankings under the two criteria of the entire schedule and retaining only the tested turn. In the author's opinion, which one is the main result and why?
  • Has any human or independent agent checked the specifications of Red's winning missions to ensure they are unambiguous?

Comments on the paper

1 comment
Area_Chair_BBqwOpenReview ↗

Reminder from AC

Dear Reviewers,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.