- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SWE-Duel evaluates coding agents through reciprocal task generation and repair, using feature-preservation and hidden bug tests. The experiments cover 11 model–harness pairs and 15 repositories.
Critical strengths
Executable tests provide concrete evaluation criteria. The fixed-solver experiments and same-model harness comparisons also provide useful controls.
Critical weaknesses
-
The current tasks are close to saturation for the strongest solvers. GPT-5.5 achieves repair success rates of 98.3% and 96.4% on contested tasks (Table 8). Table 2 shows that stronger task authors can challenge a fixed solver, but does not establish that task difficulty keeps pace as frontier solvers improve. Appendix L acknowledges this bottleneck. I find a substantial gap between the motivation of overcoming benchmark saturation and the evidence presented.
-
The self-repair admission rule may restrict task difficulty and coverage. The bank retains only tasks on which the author has already succeeded at blind self-repair. Valid tasks that expose the author’s own weaknesses can therefore be discarded. Swapping roles does not remove this selection effect. The paper does not sufficiently establish which useful hard tasks this gate excludes or how that exclusion affects the capabilities being evaluated.
Questions for authors
- Which results support continued discrimination among improving frontier solvers?
- How many otherwise valid tasks does the self-repair gate reject, and what do those tasks reveal about lost difficulty or capability coverage?
Reminder from AC
Dear Reviewers,
As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.
Thank you for your time, your thoughtful reviews, and your service to the ICLR community.
Best Regards, AC