- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Senior SWE-bench extends standard coding benchmarks to measure how coding agents perform on tasks that a senior engineer does day-to-day. The benchmark consists of 100 tasks across 12 repositories. The tasks are intended to be underspecified to mimic real-world senior software engineering day-to-day work. The agent has the freedom to choose the best approach and the benchmark is meant to capture the engineering judgement of the agents. The benchmark is designed to capture two different workflows, design-and-build and investigate-and-fix. The task success is measured across two axes: (i) Behavioral Correctness; and (ii) Taste. The best overall score (considering taste) is only 33%.
Critical strengths
- The paper identifies an important evaluation gap and the study is well motivated. Capturing engineering quality is an important direction.
- The verification agent is an important and interesting contribution. An underspecified task will produce varied solutions which cannot be captured by static tests. Hence, their solution is sound and the analysis performed by the authors is very interesting.
- The authors focus on quality control is an important contribution. The authors use multiple techniques to filter out low quality samples. The future work can use these techniques to build high quality benchmarks and even calibrate LLM-judges in their dataset pipeline.
Critical weaknesses
- Taste uses human reference solutions but there is no guarantee that the human generated solutions are the best. The judge scores between 1-5 where 5 means better than the reference but no analysis performed on the samples was judged to be better than the human reference solution.
- The taste judge infers codebase practices. Although the authors validate the judge’s score against humans, this does not establish the accuracy of the codebase practices’ inferred by the judge.
- The paper presents limited analysis. The authors can investigate and aggregate common quality dimensions responsible for agents receiving low taste scores. The authors can further include performance stratified by task type (design-and-build vs. investigate-and-fix).
- New commits and repositories such as harbor and might be heavily built using LLMs. The authors can discuss potentially AI-generated reference implementations and its effect on the benchmark and results.
Questions for authors
- Why was the verification agent not used for investigate-and-fix? Just like design-and-build, tasks under investigate-and-fix can also be fixed using different approaches.
- Given that the taste judge measures different aspects of the code semantics, additional analysis on the common failure modes will be very interesting for the community. If the authors can extend upon this, it will make the conclusions stronger.
- Can the authors expand the limited analysis and address weakness 3? I believe this will make the paper's finding more relevant.
- Given that the agents achieve high behavioral correctness, the paper can also analyze the senior-level capabilities that remain unsolved across different frontier models.
3 comments
Thank you for providing detailed rebuttal. While the rebuttal has addressed my questions, I would like to keep my original scores.
I have one more questions for the authors: Why is agent given access to the internet during evaluation?
I believe the answer leakage observed is largely a set up issue and authors can improve the set up by setting "--network none" for docker container while running evaluation and grading.
Thank you for following up so quickly on our response. In response to your question, the solving agent runs under a default-deny allowlist of about 30 hosts and does not have full internet access (Appendix B). Part of the design of the benchmark is to assess agents' abilities to set up environments (e.g. installing dependencies, starting services) and researching technologies, which requires (restricted) internet access. The allowlist includes only resources like package registries, toolchain installers, and official language references. It intentionally excludes the primary leak vectors like GitHub. The allowlist blocks over 80% of solution-retrieval attempts, and the remaining successful attempts can be reliably detected by post-hoc trajectory analysis and voided, so no hacked solution counts toward any score (§5, Appendix E.3). We will clarify the allowlist configuration in the main text and we are also developing a fully offline variant.
Thank you for the constructive review, and for recognizing the evaluation gap, the verification agent, and the quality-control pipeline.
On whether the reference solutions are best: in this benchmark, we primarily measure whether an agent reaches a real-world bar, i.e. code that is high quality enough such that the repo’s maintainers would accept it. As a result, the reference solutions are meant to represent this bar, as opposed to a single, globally optimal solution. The relative taste scores then reward a solution for being close to this bar, and are relatively insensitive to whether a patch is perceived to be better than the reference Eq. 1 gives a 5 almost no extra credit beyond a 4, and such patches are rare with only 0.5% receiving a 5 from both judges.
On inferred codebase practices, we directly audited and found that they are largely accurate. As a clarification, the judge does not enumerate practices up front; it checks whether the patch aligns with the observed patterns in the code around it, because written conventions are often not followed in practice. On the 119 annotation patches, we had the two taste judges record every practice behind each practice-alignment score, with code evidence and how widespread the practice is. There were 2,216 practices recorded, an average of 1.86 per rubric dimension. An independent auditor from a third model family (Gemini 4 Argon) checked each one against the base commit, and we also performed human spot-checks.
We will include the previous analysis in the Appendix. Even with these results, the sensitivity of the final taste score to individual practice assessments is relatively low. The general taste score accuracy is best understood via agreement with human annotators.
Analyzing the common quality dimensions responsible for agents receiving low taste scores yields interesting insights, and we are adding this analysis to the Appendix. Even among correct patches, the dimensions that most commonly scored ≤2 are solution approach (45%), craftsmanship (44%), pattern adherence (36%), abstraction level (22%), and minimality (21%). Surface style, library use and documentation are rare (3–7%). These ≤2 rates fall with overall performance tier. Additionally, profiles differ between models. Opus 5 has the fewest solution approach failures but most often over-builds (minimality 34%), while Sonnet 5 has a very high rate of ≤2 on documentation (30%).
Given the behavioral differences between models, we also agree with breaking out the per-task-type performance into their own leaderboards. Overall, investigate-and-fix is harder: the top score is 29, against 38 for design-and-build. The ranking is broadly stable, but some profiles flip. Sonnet 5 is in the top tier for design-and-build and in tier 2 for investigate-and-fix, while GPT-5.6 Terra is the reverse. This will be added to the Appendix.
AI-generated reference implementations are a relevant concern, and we were careful to ensure all source PRs are primarily human-designed by repo maintainers or frequent committers. We re-audited all 100 source PRs. 0 show evidence of being fully AI-designed, implemented, and reviewed. 34 disclose AI assistance, mostly Claude Code co-authorship. Every one was reviewed and merged by a human account, and 82% show review-driven iteration via PR discussion between humans. Excluding them leaves tier 1 unchanged (ρ = 0.996). Furthermore, Claude-based solving agents gain no advantage on Claude-assisted references: Anthropic models' lead over other models is smaller on those tasks than on the rest (6.2 vs 9.5 points; difference 3.4 [−1.9, 8.6]). We will include this analysis in the Appendix.
On using the verification for investigate-and-fix tasks: investigate-and-fix tasks are designed so that no near-optimal fix would plausibly change system interfaces [§4.1]. The contract is the existing observable behavior at existing entry points, making them suitable for pre-written verifier tests (i.e. no need for adaptation by the verification agent). As evidence, the pre-written verifiers agreed with expert correctness labels on all 40 investigate-and-fix patches in our validation set (0/22 false rejects, 0/18 false accepts).
To provide additional analysis of senior-level capabilities that remain unsolved, we’re adding an Appendix figure with solve rates for each skill in our task taxonomy. Across the top-tier models, Overall is lowest on event-driven design (22%), dependency research (24%), and logic and fault localization in bug fixing (25–26%). It is highest on architectural refactoring (49%) and state management (45%).