acceptodds
Under review as a conference paper at ICLR 2027

Senior SWE-bench: Evaluating Coding Agents as Senior Software Engineers

Abstract

Coding agents are expected to perform like senior engineers: they are tasked with building features from underspecified requirements, investigating and fixing bugs from user-reported symptoms, and writing high-quality code that meets broader codebase standards. However, popular benchmarks evaluate agents like “junior” software engineers, with highly specified instructions and a focus on behavioral correctness alone. This gap is a consequence of standard grading approaches: pre-written verifiers require full interface specification while LLM-as-a-judge approaches give up the reliability of executing code. Neither can reliably grade senior-level work, which admits many valid solutions that differ in quality. We close the gap with two mechanisms: a verification agent that writes verifier tests adapted to each solution, and a taste judge that assesses code quality in a grounded manner. Using these techniques, we built Senior SWE-bench, a benchmark of 100 tasks (50 public and 50 held-out to resist contamination) drawn from real senior engineering work. Tasks are “design-and-build” (e.g., new features addressing user stories) or “investigate-and-fix” (e.g., diagnosing and solving complex bugs), and instructions are naturally underspecified (on average, 26% of the median length of those in SWE-Bench Pro). Evaluating 46 frontier agent configurations with these grading mechanisms, we find that the best behavioral-correctness pass@1 is 62.1%, while the best overall pass@1 (which also considers taste) is only 33.3%. More broadly, we show that for frontier agents, correctness and taste rankings often diverge, and that Senior SWE-bench tasks can reveal trends in model behavior.

Primary area:
datasets and benchmarks
Keywords:
coding agents, software engineering, benchmark, code quality, taste

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JrPkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Senior SWE-bench extends standard coding benchmarks to measure how coding agents perform on tasks that a senior engineer does day-to-day. The benchmark consists of 100 tasks across 12 repositories. The tasks are intended to be underspecified to mimic real-world senior software engineering day-to-day work. The agent has the freedom to choose the best approach and the benchmark is meant to capture the engineering judgement of the agents. The benchmark is designed to capture two different workflows, design-and-build and investigate-and-fix. The task success is measured across two axes: (i) Behavioral Correctness; and (ii) Taste. The best overall score (considering taste) is only 33%.

Critical strengths
  1. The paper identifies an important evaluation gap and the study is well motivated. Capturing engineering quality is an important direction.
  2. The verification agent is an important and interesting contribution. An underspecified task will produce varied solutions which cannot be captured by static tests. Hence, their solution is sound and the analysis performed by the authors is very interesting.
  3. The authors focus on quality control is an important contribution. The authors use multiple techniques to filter out low quality samples. The future work can use these techniques to build high quality benchmarks and even calibrate LLM-judges in their dataset pipeline.
Critical weaknesses
  1. Taste uses human reference solutions but there is no guarantee that the human generated solutions are the best. The judge scores between 1-5 where 5 means better than the reference but no analysis performed on the samples was judged to be better than the human reference solution.
  2. The taste judge infers codebase practices. Although the authors validate the judge’s score against humans, this does not establish the accuracy of the codebase practices’ inferred by the judge.
  3. The paper presents limited analysis. The authors can investigate and aggregate common quality dimensions responsible for agents receiving low taste scores. The authors can further include performance stratified by task type (design-and-build vs. investigate-and-fix).
  4. New commits and repositories such as harbor and might be heavily built using LLMs. The authors can discuss potentially AI-generated reference implementations and its effect on the benchmark and results.
Questions for authors
  1. Why was the verification agent not used for investigate-and-fix? Just like design-and-build, tasks under investigate-and-fix can also be fixed using different approaches.
  2. Given that the taste judge measures different aspects of the code semantics, additional analysis on the common failure modes will be very interesting for the community. If the authors can extend upon this, it will make the conclusions stronger.
  3. Can the authors expand the limited analysis and address weakness 3? I believe this will make the paper's finding more relevant.
  4. Given that the agents achieve high behavioral correctness, the paper can also analyze the senior-level capabilities that remain unsolved across different frontier models.
3 comments

Thank you for the constructive review, and for recognizing the evaluation gap, the verification agent, and the quality-control pipeline.

  • On whether the reference solutions are best: in this benchmark, we primarily measure whether an agent reaches a real-world bar, i.e. code that is high quality enough such that the repo’s maintainers would accept it. As a result, the reference solutions are meant to represent this bar, as opposed to a single, globally optimal solution. The relative taste scores then reward a solution for being close to this bar, and are relatively insensitive to whether a patch is perceived to be better than the reference Eq. 1 gives a 5 almost no extra credit beyond a 4, and such patches are rare with only 0.5% receiving a 5 from both judges.

  • On inferred codebase practices, we directly audited and found that they are largely accurate. As a clarification, the judge does not enumerate practices up front; it checks whether the patch aligns with the observed patterns in the code around it, because written conventions are often not followed in practice. On the 119 annotation patches, we had the two taste judges record every practice behind each practice-alignment score, with code evidence and how widespread the practice is. There were 2,216 practices recorded, an average of 1.86 per rubric dimension. An independent auditor from a third model family (Gemini 4 Argon) checked each one against the base commit, and we also performed human spot-checks.

    • 81.9% of cited practices are verified, representative, and correctly evidenced.
    • The judge's decision on whether the patch follows a practice is right 92.1% of the time.
    • The most common failure mode was the judge using code from the solving agent’s patch as pattern evidence; this was generally infrequent but is worth improving in benchmark updates.

    We will include the previous analysis in the Appendix. Even with these results, the sensitivity of the final taste score to individual practice assessments is relatively low. The general taste score accuracy is best understood via agreement with human annotators.

  • Analyzing the common quality dimensions responsible for agents receiving low taste scores yields interesting insights, and we are adding this analysis to the Appendix. Even among correct patches, the dimensions that most commonly scored ≤2 are solution approach (45%), craftsmanship (44%), pattern adherence (36%), abstraction level (22%), and minimality (21%). Surface style, library use and documentation are rare (3–7%). These ≤2 rates fall with overall performance tier. Additionally, profiles differ between models. Opus 5 has the fewest solution approach failures but most often over-builds (minimality 34%), while Sonnet 5 has a very high rate of ≤2 on documentation (30%).

    Given the behavioral differences between models, we also agree with breaking out the per-task-type performance into their own leaderboards. Overall, investigate-and-fix is harder: the top score is 29, against 38 for design-and-build. The ranking is broadly stable, but some profiles flip. Sonnet 5 is in the top tier for design-and-build and in tier 2 for investigate-and-fix, while GPT-5.6 Terra is the reverse. This will be added to the Appendix.

  • AI-generated reference implementations are a relevant concern, and we were careful to ensure all source PRs are primarily human-designed by repo maintainers or frequent committers. We re-audited all 100 source PRs. 0 show evidence of being fully AI-designed, implemented, and reviewed. 34 disclose AI assistance, mostly Claude Code co-authorship. Every one was reviewed and merged by a human account, and 82% show review-driven iteration via PR discussion between humans. Excluding them leaves tier 1 unchanged (ρ = 0.996). Furthermore, Claude-based solving agents gain no advantage on Claude-assisted references: Anthropic models' lead over other models is smaller on those tasks than on the rest (6.2 vs 9.5 points; difference 3.4 [−1.9, 8.6]). We will include this analysis in the Appendix.

  • On using the verification for investigate-and-fix tasks: investigate-and-fix tasks are designed so that no near-optimal fix would plausibly change system interfaces [§4.1]. The contract is the existing observable behavior at existing entry points, making them suitable for pre-written verifier tests (i.e. no need for adaptation by the verification agent). As evidence, the pre-written verifiers agreed with expert correctness labels on all 40 investigate-and-fix patches in our validation set (0/22 false rejects, 0/18 false accepts).

  • To provide additional analysis of senior-level capabilities that remain unsolved, we’re adding an Appendix figure with solve rates for each skill in our task taxonomy. Across the top-tier models, Overall is lowest on event-driven design (22%), dependency research (24%), and logic and fault localization in bug fixing (25–26%). It is highest on architectural refactoring (49%) and state management (45%).

Thank you for providing detailed rebuttal. While the rebuttal has addressed my questions, I would like to keep my original scores.

I have one more questions for the authors: Why is agent given access to the internet during evaluation?

I believe the answer leakage observed is largely a set up issue and authors can improve the set up by setting "--network none" for docker container while running evaluation and grading.

Thank you for following up so quickly on our response. In response to your question, the solving agent runs under a default-deny allowlist of about 30 hosts and does not have full internet access (Appendix B). Part of the design of the benchmark is to assess agents' abilities to set up environments (e.g. installing dependencies, starting services) and researching technologies, which requires (restricted) internet access. The allowlist includes only resources like package registries, toolchain installers, and official language references. It intentionally excludes the primary leak vectors like GitHub. The allowlist blocks over 80% of solution-retrieval attempts, and the remaining successful attempts can be reliably detected by post-hoc trajectory analysis and voided, so no hacked solution counts toward any score (§5, Appendix E.3). We will clarify the allowlist configuration in the main text and we are also developing a fully offline variant.

Reviewer_WS3xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work propose to assess the coding taste of existing LLMs. The authors build a benchmark with a sophisticated pipeline with quality control, including 100 tasks from 12 diverse repositories. For reliable and taste-oriented grading, a verification agent and a taste judge are designed to assess both code correctness and quality. Moreover, the tests are specifically tailored for each solution without a requirement for predefined interfaces. Frontier models are evaluated including Claude, GPT, Kimi, GLM, etc.

Critical strengths
  1. The design of grading provides a good option for underspecified instructions, eliminating the need for defining exact interfaces.
  2. The task creation pipeline introduces good quality control with multiple stages of checks including human experts.
Critical weaknesses

The tastes are generally subjective and hard to measure. The reviewer is not convinced of the value of the benchmark and assessment.

  • Even human experts show low consistency 50% with each other. The reviewer noticed that 90% of the distinctions are within 1, but it may mainly due to a 15 grading range. Within a 1100 grading range, the gap is 20%. There should be many ambiguous or controversial places. It raises questions about the meaning of metrics and their differences. How can one interprete the result that one model obtains 5 more scores than the other?
  • The value of the taste assessment in this work is not convincing. Does the taste score help distinguish different levels of maintainablity? For example, how will the scores correlate with bugs or difficultiies of implementing new features in future development?
  • Currently the evalution seems to be performed on the "raw" models. What if the model or agent is provided with a skill, AGENTS.md or clear instructions? If the model can well write high quality code with a skill, current assessment only provides a very limited view.
  • For the grader, if it can be experimented with RL, its value will be more prominent.
Questions for authors

The authors could also consider how to make the pipeline more suitable for scalable agentic task generation.

3 comments

Thank you for your review and for highlighting the value of grading mechanisms for underspecified instructions without fixed interfaces, and for recognizing the multi-stage quality control. We address your main concern of whether taste can be measured meaningfully with new analyses and examples.

  • We believe that senior developers have a strong and consistent sense of taste and code quality on the aggregate. Codifying and operationalizing this sense of taste is critical to the development of next-generation coding agents. Our goal in this work is to establish a clear definition for grounded assessment of taste in software engineering, especially as it relates to code quality. Every score is relative to a concrete, observable, and real-world bar: the repository's own observed practices and a reference patch its maintainers reviewed and merged. Furthermore, taste is broken down into 10 expert-designed dimensions that relate to codebase health and maintainability, which senior engineers often focus on during review. The points below show this bar is measurable and identifies real problems in solutions beyond behavioral correctness.

  • Thanks for raising the point on consistency between experts, as it will help us clarify the presentation in the paper. When we look at the taste scores as they contribute to Eq. 1 (each mapped to a 0-1 credit), two experts’ scores agree within 0.05 on practice alignment for 89% of patches and relative taste for 70%. Their final credit correlates at Spearman ρ = 0.82. The signal from the taste assessment is best viewed through this lens, i.e. as scoring whether a patch is above a quality bar (via the sigmoid), rather than where it falls on a fine scale. We will add this to Table 2.

    This interpretation also addresses interpretability of the scores. The Overall measurement is approximately the share of tasks solved both correctly and to a sufficient code quality standard, so a 5-point gap means about 5 more of 100 tasks a reviewer would assess as meeting both sufficient correctness and code quality.

    To build intuition for where Taste score helps discriminate, it can also be informative to look at examples where the agent passed on correctness, but had a low quality score. This demonstrates what the two scores capture together that just the standard correctness score does not. We have included the following two examples in the subsequent comment and will further add these to the Appendix.

  • On maintainability, the taste rubric is constructed to reflect how a senior reviewer would distinguish different levels of maintainability. Code that fits the project's patterns, reuses its abstractions, and avoids needless structure is easier to understand, review, and extend. Higher taste therefore is meant to correspond to more maintainable code and faster feature development over time. The detailed taste judge rubric operationalizes this directly, for example:

    • The craftsmanship dimension asks whether a change "leaves the codebase easier to work in going forward", with "no needless coupling, foot-guns, or maintenance burden".
    • The approach quality dimension asks whether a solution "would hold up as requirements evolve".
    • The library usage, pattern adherence, abstraction, and hygiene dimensions target redundancy, divergence, over-engineering and band-aid fixes.

    This mirrors real code review, where most issues reviewers raise concern evolvability rather than functional defects [Bacchelli & Bird, ICSE 2013; Mäntylä & Lassenius, TSE 2009].

  • We study exactly the question of what happens when providing the agent with clear instructions. This is what was measured by the additional guidance experiment [§5]. We observed that Behavioral Correctness increased significantly (24.6 points on average) while Taste barely increased (2.3 point avg). This demonstrates that underspecified tasks challenge models, but that taste and code quality are more inherent to model behavior.

    We focus on a fair assessment of a model’s inherent engineering capabilities and judgement, which is why we evaluate with one fixed harness rather than experimenting with different skills or other guidance.

  • We agree with RL results demonstrating value. Using the grader as an RL reward is in progress, and early results show meaningful gains: training on just 50 tasks yields 50% improvement in pass@1 for Qwen3.6-35B.

  • On scalable generation, we have experimented with agentic approaches for each stage of the pipeline except expert review: candidate mining, task authoring, trial-based quality checks, and repair. Even where some quality can be sacrificed for greater scale, we still recommend human review to preserve quality and realism, though it can be reduced to a smaller sample when scale matters most. Generation then trades off three things: yield (how many authored tasks are used vs. abandoned), quality (how many weaker tasks get through), and speed, with human review as the bottleneck.

Following up on our response above, we are including two examples where the agent passed on correctness but had a low Taste score:

  • Task: auth refactor in gitea. For the reference solution, the maintainers attach auth inside the existing middleware pipeline. In the task trial, the agent tracks per-route auth via global state using a mutex, a secondary router, and a test-only hook that resets state leaked between tests. It passes all behavioral tests but would be rejected by any maintainer for poor production engineering practices. The patch is half the reference's size, so the bloat score doesn’t penalize it. The taste judge, however, scores pattern adherence, abstraction, approach, hygiene and craftsmanship at 2, noting that the registry "can silently get out of sync with the real router", and the patch has a very low taste score.
  • Task: dependency cycle detection in Turborepo. For the reference solution, the maintainers use the tarjan_scc method in petgraph, a library used throughout the repo for graph operations. In the task trial, the agent hand-rolls a 94-line Tarjan's strongly-connected-components routine, and adds an opt-out from task-graph validation that is effectively dead code. Again, it passes all behavioral tests but would be rejected by a senior reviewer for poor maintainability and over-engineering. Both the size gate (3.4×) and the taste judge (pattern adherence and library usage at 2) produce a low score.

Thank you again for your review. We revised the paper based on your feedback, and included aspects like validation of taste credit and illustrative examples for how the grading mechanisms work in concert. Please let us know if you have any additional feedback!

Reviewer_TXXnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a 100-task coding benchmark (half public, half held out) from merged pull requests in 12 repositories across five languages. Half the tasks ask the agent to design and build a feature, half to investigate and fix a bug. Submissions are graded in two ways. A verification agent takes expert-written behavioural "stories", writes test scripts that fit whatever design the submission chose, has a second model screen those scripts for common defects, and then runs them. A taste judge, a panel of two LLMs, scores ten code-quality dimensions, which are combined with a code-size ratio and a rubric fraction into a single credit through a hand-set formula. The headline score multiplies correctness by taste. The authors validate the verifier against expert labels on 60 patches and against pre-written tests and a plain LLM judge, validate the taste rubric against two engineers on 119 patches, and run 18 models three times each. The best model solves 62% of tasks correctly, but the best overall score is 33%.

Critical strengths

The verification agent design is a strength of this paper. Grading underspecified feature tasks is a real problem: pre-written tests fail good solutions that chose a different design, and plain LLM judges wave through plausible-looking code. The recipe design separates what the solver is told from how it is checked, and the validation shows each piece matters: static tests reject 26% of correct solutions, the recipe-free agent accepts 45% of wrong ones, while the full agent sits at 19% false accept and 4% false reject. Planting defects in its scripts and re-running it three times addresses the obvious objections to having a model in the grading loop (though this is still problematic to some extent).

Critical weaknesses
  • The taste number is never validated. Table 2 checks the ten rubric dimensions, not the final credit that Equation 1 produces from them. That formula has a size penalty strong enough to halve the credit for a patch twice the reference length, and the thresholds and temperature are just picked. So an agent that writes tests gets punished on taste, and we never find out how much of the gap between models is that penalty versus actual code quality. The expert agreement in Table 2 also compares different things, 714 per-dimension pairs for the experts and 119 whole-patch pairs for the judge.

  • A few things missing from the leaderboard could move it. How often does the verification judge discard a trial, per model? If that differs, the pass rates are over different task sets. All the graders are Claude or OpenAI models and the top rows are Claude; is there any check with a third-party judge? There are no confidence intervals, and with 95 tasks a 33% rate has about a five-point standard error, so the top five rows are a tie. And nothing is said about training cutoffs for PRs merged after February 2026.

Questions for authors
  • What fraction of trials did the verification judge discard, overall and per model, and what does the leaderboard look like if discards count as failures? (Clarification)
  • Section 4.2 says combining pre-written tests with the agent "decreases the false reject rate to 0.11", but Figure 4 shows 0.04 for the agent alone and adding tests can only raise it. Is 0.11 the false accept rate? (Clarification)
  • How are 714 expert-versus-expert pairs obtained from 119 patches with five dimensions per rubric, and is the judge compared per dimension or per rubric median? (Clarification)
  • Please report taste with the size term removed and the sensitivity of the ranking to the formula's temperature. (Suggestion)
  • Which models were used in the "agent trials" step of task construction, and was any task revised based on a trial from a model later evaluated? (Clarification)
2 comments

Thank you for the careful and technical review, and for recognizing the verification agent's design and its validation as a strength. We agree with your feedback and suggestions, and have updated our analyses as discussed below.

  • Thank you for highlighting that we didn’t include a validation of the taste score. We now validate the final credit from Eq. 1 directly. We computed Eq. 1 from the expert dimension scores and from the judge’s on the same patches. The judge's credit correlates with the expert consensus at Spearman ρ = 0.76 [0.64, 0.85] (0.80 on correct patches), close to the agreement between two experts (ρ = 0.82 [0.73, 0.89]). We’re adding a new sentence in §4.2 and a table in the Appendix.

  • As a clarification on bloat: test, documentation, and other non-behavioral file changes are not penalized by the bloat term. An LLM-based file classifier identifies and removes these changes from both the agent patch and reference before counting lines of code. We re-audited the results of the classifier and found 0 test files leaked through in 150 classifier output - confirming that an agent that writes tests does not get punished on taste. We’ll update §3 to state this explicitly.

    We include a bloat term to reflect the maintainability/human legibility cost of unnecessary code and complexity - a senior reviewer would push back on a PR 3-4× as long as it needs to be. However, the leaderboard ranking is relatively insensitive to the specific choice of the term. We experimented with varying the following in Eq. 1:

    • for all terms, temperature w ∈ {0.125, 0.5, 1}
    • for the a bloat term, adding a buffer of a few lines so that a high multiple of a very small reference change isn’t penalized
    • removing the bloat term altogether

    Every variant keeps Spearman ρ ≥ 0.96 with the reported ranking, and the same four models score highest under all of them. We will include these notes on sensitivity in the Appendix.

  • Thank you for asking about expert-versus-expert pair counts. There was a typo in the paper, which said two human SWEs annotated the 119 patches. It was four human SWE annotators, and their annotations were compared pair-wise on the per-rubric median. This gives 119 x C(4, 2) = 714 comparisons. We’ll make sure to correct this.

  • On verification agent review judge behavior, the discard rate is 2.5% of grading attempts overall (0–5.6% per model, and at most 1.4% for every top-tier model). A discarded attempt is re-run, so every model is scored on the same task set. We interpret the second question slightly differently: a discard means the grader's own verdict was untrustworthy, not that the solution failed. Counting discards as failures would penalize models for grader errors. We will include a note on discard rates in the Appendix.

  • We performed an analysis with a third-party taste judge (Grok 4.6 from xAI) and scores were nearly unchanged (κ 0.86–0.88). We also found taste judge models do not significantly favor their own family: own-family score contrasts are +0.07 [−0.14, 0.28] for Anthropic, +0.05 [−0.14, 0.24] for OpenAI, and −0.11 [−0.40, 0.15] for xAI.

  • Thanks for pointing out that Table 3 would benefit from confidence intervals; we agree. We will now report avg@3 with paired task-bootstrap 95% CIs and add tiers in addition to ranks. A model joins a tier unless it is significantly below the tier leader (Holm-corrected, family-wise α = 0.10). We updated the chosen effort level for each model accordingly. The top four models (Fable 5.1, Fable 5, Opus 5, GPT-5.6 Sol; 28–35) are statistically tied and separated from tier 2 (22–26). We repeated this analysis for each task type as well since models differ in their performance profiles. On Design-and-Build, Sonnet 5, Grok 4.6, and Opus 4.8 join tier 1. On investigate-and-fix, GPT-5.6 Terra joins tier 1.

  • On training cutoffs, source PRs were merged between February and June 2026 with 67% in April 2026 or later. For 9 of the 12 models with published or reported cutoffs, at least 94% of tasks were merged after the cutoff. If we misunderstood this feedback, please let us know.

  • On the 0.11 figure, yes thank you, it is the false accept rate. Corrected in §4.2.

  • For agent trials during construction, we rotated among Opus 4.8, Sonnet 5, Grok 4.5, and GPT-5.5. Most tasks then received at least a small revision, generally for consistency or test completeness. Those revisions fixed grading (e.g. false rejects or ambiguous specifications). No task was tuned to be harder or easier for a particular model.

Thank you again for your review. We revised the paper based on your feedback, and included aspects like validation of taste credit, confidence intervals and tiers for the leaderboard, sensitivity analysis for the taste terms, and more. Please let us know if you have any additional feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We again want to thank all of the reviewers. We are encouraged that they found the overall evaluation gap we addressed, the technical approach to adaptive grading for underspecified tasks, and focus on task quality control to be strengths. Their feedback has improved the paper and we have uploaded a revised PDF (changes are highlighted in blue). The most significant changes include:

  • Additional validation of taste credit in §4.3 and App. D.2
  • Adding confidence intervals and statistical tiers to the leaderboard
  • Expanded analysis including per-task-type leaderboards, failure rates by SWE skills, and challenging aspects of tasteful code
  • Examples to illustrate how different grading mechanisms work in concert

We're happy to update the camera-ready version in response to any additional discussion or feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.