acceptodds
Under review as a conference paper at ICLR 2027

swespoke: Declare What You Want, Get a Bespoke Software Engineering Benchmark

Abstract

Coding benchmarks must rank agents, expose where they fail, and tell developers which agent to trust with their tasks. Early benchmarks were fixed task sets that humans defined, and current benchmarks mine tasks from an existing repository. Mining is slow and expensive, and at the end, users (software-engineering teams) get every task the repository yields. This process never asks what work users care about or which properties they want in tasks. Users, not benchmark authors, must be able to declare task properties. These declarations let users test the capability they care about, raise difficulty as agents improve, build an RL curriculum from easy to hard, and pay only for tasks that have the declared properties. These needs differ across users and change over time. swespoke lets users declare the properties they want in tasks, and a query optimizer turns the declaration into an optimized plan that builds the benchmark to reduce execution time and cost. We call the process task tailoring. Unlike mining, tailoring checks the declaration before it builds a task, so selection and creation run as one step and users pay for no task they throw away. Each task comes with a runnable environment, a build, and tests, and swespoke verifies the task by running its tests. We also build NDI, a navigation difficulty index, as an example of a user-defined property. NDI estimates how hard it is to find the code a task must change. Users compose benchmarks at a chosen NDI level. Compared with the baselines, swespoke speeds up task creation by 7.27 to 12.48×, reduces LLM expense by 2.39 to 4.85×, and accelerates measurement collection by 14.43 to 19.94×. The average coding-agent pass rate falls from 68% overall to 46% on the hardest 10% of tasks by NDI.

Primary area:
datasets and benchmarks
Keywords:
Software Engineering Benchmark, Benchmark Customization, Task Selection, Task Difficulty Measurement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_68qzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper introduces a framework for creating tailored SWE-bench-style tasks from existing repository histories. It lets users declare measurements, conditions, and actions to select or revise tasks based on desired properties and difficulty. The framework optimizes filter order and shares computations to avoid unnecessary construction work

  • Authors run experiments on RocksDB, Polars, and DuckDB to produce 223 tasks, reporting 7-12x faster construction and 2-4× lower LLM expense than a simpler baseline pipeline. The authors also propose a navigation difficulty index (NDI) for selecting harder tasks as an example usage of the framework, reducing average agent pass rate from 68% overall to 46% on the hardest 10%

Critical strengths
  • The paper addresses a practical issue of task construction efficiency for SWE Agent tasks. Applying pre-filtering is a very reasonable practices that practitioners adopt to produce desirable tasks.

  • The measurement-condition-action framework provides a flexible way to tailor SWE Agent tasks construction pipelines without big refactoring for different requirements.

Critical weaknesses
  • Early filtering and shared computation are practical ways to reduce task-construction costs, but the paper does not clearly establish methodological novelty beyond established filter ordering, batching, and computation reuse. The contribution appears primarily to be an engineering framework

  • The reported cost reductions are not accomplished by proper comparison of task quality, such as task fairness, test coverage, the noise ceilings of the benchmark, etc. It is also unclear whether the baseline applies identical final selection criteria. Without matched output requirements or independent quality assessment, the efficiency results are difficult to interpret

  • The framework focuses on single PR, SWE-bench styled tasks, while the industry have moved on to more broader task ranges and diverse classifiers (e.g. DeepSWE, SWE Atlas). The framework feels more relevant to the construction of training data instead of benchmarking, yet quality signal on the produced tasks as training data is missing

Questions for authors
  • Does the simpler baseline apply the same selection criteria as swespoke after task construction, including the multiple-source-file requirement, or does it retain tasks that swespoke rejects?

  • Could the authors assess the quality of swespoke-generated tasks, and compare against an established task-construction method such as SWE-smith?

2 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer 68qz (Part 1 of 2)

Thank you for your constructive feedback. We apologize for the delay in our response. Our lead author had a medical emergency, so we were unable to respond earlier in the rebuttal period. We hope you will still consider our responses and the revised paper. We would be happy to answer any follow-up questions.

W1: The technical problem that swespoke resolves is how to automatically apply a user’s selection requirements during task creation while minimizing the cost of producing the requested task set. This is challenging because task selection and task creation depend on each other. For example, some requirements can be checked directly from repository history, while others require building an environment and running tests. Without a general way to integrate selection into task creation, users have to manually decide where to check each new requirement and how to act on the result.

Our contribution is to formalize task selection during task creation as a constrained cost-minimization problem and to design the abstraction of filters that enables users to specify any selection requirements without manually modifying the construction pipeline. None of the task construction approaches we examined, including SWE-bench-Live, SWE-Factory, and Repo2RLEnv, describes a cost-based optimizer for automatically applying user-defined selection criteria during task creation. We also agree that early pruning and batching are established optimization principles. The novelty lies in modeling how pruning and batching affect the cost of creating the requested task set and using those estimates to decide when to apply each filter and which measurements to compute together.

We clarified these contributions in Sections 2.1 and 2.4 of the revised paper.

W2: For task selection criteria: we would like to clarify that both swespoke and the baseline apply the same final selection criteria and return the same 202 RocksDB tasks, 341 Polars tasks, and 488 DuckDB tasks (Table 2). The only difference is that swespoke automatically applies the file-count, change-location, and assertion-count filters, as well as the description–test alignment check, to prune candidates during construction, whereas the baseline applies those three filters and excludes flagged tasks after construction. The savings mainly come from skipping LLM assessments, environment creation, and verification for candidates rejected early. We clarified the setup in Section 3.1 and the results in Figure 3 and Table 2.

For task quality: we would also like to clarify that swespoke can use any automated task-quality check as a measurement, so users can add or replace a checker without having to manage how it is applied during task selection and creation. Designing a reliable quality checker is therefore orthogonal to the task selection problem that swespoke addresses. In future work, we will systematically study how to assess task quality, including task fairness, test coverage, and benchmark noise ceilings.

That said, we agree that it is important to validate the quality of the tasks used in our experiments. We added two task-quality assessments in Section 3.3 and Appendix E. First, we compared swespoke’s description–test alignment checker and the SWE-bench-Live prompt against independent human ratings on the same 1,699 tasks reviewed for SWE-bench Verified (Chowdhury et al., 2024). swespoke marked 455 tasks as aligned, of which 324 (71.2%) had tests rated acceptable in the published annotations. The corresponding share for the SWE-bench-Live prompt was 553 of 1,027 (53.8%). Among the tasks retained by each checker, 10.8% received the most severe rating for swespoke, compared with 18.2% for SWE-bench-Live (Table 9).
Second, on 210 of the 223 swespoke tasks (94.2%), at least one of six coding-agent configurations produced a submission that the grader accepted (Table 8). Solvability matters because flawed tests can reject correct solutions, as OpenAI found in its audit of 138 SWE-bench Verified tasks that o3 struggled to solve consistently (OpenAI, 2026). Together, these assessments show that a larger share of the tasks retained by swespoke met the human reviewers’ standard for test fairness and that most swespoke tasks in our experiments were solvable.

AuthorsOpenReview ↗

Authors’ Response to Reviewer 68qz (Part 2 of 2)

W3: On single-PR tasks: we agree that swespoke currently creates each task from a single PR. We appreciate the reviewer’s observation that software engineering evaluations are expanding to cover a broader range of work. We updated our future work discussion to include these long-horizon and more diverse engineering tasks, alongside our existing plan to automate multi-PR task creation. However, we also observe that single PRs still offer a practical way for engineering teams to create evaluation tasks from their own repositories, as shown by recent work at Databricks, Red Hat, and ServiceNow (Gaba et al., 2026; Bende, 2026; Hinckley et al., 2026). Also, as our evaluation shows, current coding agents still fail on these specialized tasks, giving engineering teams feedback they can use to improve their agents.

On the utility of swespoke tasks: we added a RocksDB case study in Section 3.4 to illustrate how swespoke can help users evaluate and improve coding agents on work drawn from their own repositories. We tested whether feedback from RocksDB tasks could improve Claude Code’s performance on changes that span multiple source files. We used swespoke to create tasks from RocksDB pull requests that modified at least two source files. We split 100 such tasks into 50 training, 20 validation, and 30 test tasks with similar TDI distributions. We then used GEPA to refine Claude Code’s repository instructions (CLAUDE.md) from its execution traces and test results. On the 30 RocksDB test tasks, Claude Code’s pass rate rose from 66.7% (20/30) without the added repository instructions to 76.7% (23/30) with the revised CLAUDE.md, a gain of 10 percentage points.
This case shows how users can use failures on tasks from their own repository to evaluate and refine their coding agents. As new pull requests arrive, users can periodically rerun their customized swespoke declaration to select new tasks and update a customized CLAUDE.md for each repository.

References:
Vinay Gaba, Ankit Mathur, Rishabh Singh, Patrick Wendell, and Matei Zaharia. July 8, 2026. Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase. Databricks Blog.
Rounak Bende. October 21, 2026. Red Hat enterprise-specific benchmarks for coding agents: Why we built RH SWE-bench. Red Hat Emerging Technologies.
Chris Hinckley, Andrea Tan, Sherry Han, Estelle Hu, Meiru Han, Zianyi Wang, Om Babar, Erin Lafrenz, and Aaliyah Brown. November 4, 2026. Benchmarking Coding Agents on the Now Platform Codebase. ServiceNow Developer Blog.

Q1: Yes. Both pipelines apply the same final selection criteria and return the same 202 RocksDB tasks, 341 Polars tasks, and 488 DuckDB tasks (Table 2). As discussed in W2, the only difference is that swespoke automatically applies filters and prunes candidates during construction, while the baseline applies the non-built-in filters after construction. Most of the savings come from skipping LLM assessment, environment creation and verification for candidates rejected early. We clarified this comparison in Sections 3.1 and 3.2.

Q2: As discussed in W2, we added two task-quality assessments in Section 3.3 and Appendix E. First, we compared swespoke’s task description-test alignment checker with the SWE-bench-Live prompt on the same 1,699 tasks reviewed for SWE-bench Verified. According to the published human annotations, 71.2% of the tasks retained by swespoke had acceptable tests (324/455), compared with 53.8% of those retained by the SWE-bench-Live prompt (553/1,027) (Table 9). Second, on 210 of the 223 swespoke tasks (94.2%), at least one of six coding-agent configurations produced a submission that the grader accepted (Table 8). Together, these assessments support using swespoke’s checker to select tasks with fair tests and show that most of the generated tasks were solvable.

SWE-smith creates a different type of task by introducing bugs that cause existing tests to fail, making a direct comparison with swespoke’s PR-based tasks difficult. We instead chose SWE-bench-Live as an established task-construction pipeline and compared its prompt and swespoke’s checker against published human judgments on the same types of tasks. We agree that assessing task quality is important. Because swespoke works with any automated quality checker, checker design is orthogonal to the task selection problem that swespoke addresses. We updated the conclusion to include a systematic study of methods for assessing task quality as future work.

Reviewer_WRZVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents swespoke, a framework for constructing customized software engineering benchmarks from repository change histories according to user-declared properties. Users specify measurements, conditions, and processing actions through a Measurement–Condition–Action interface, while the system reduces construction costs through early filtering and shared computation. The authors also introduce a navigation difficulty index, NDI, to demonstrate task selection at different difficulty levels. The experiments cover three repositories and three agent configurations.

Critical strengths

The Measurement–Condition–Action interface separates property computation, selection rules, and task processing, facilitating composition and reuse.

Integrating filtering before environment construction and verification aligns with the actual cost structure of software engineering task construction. The stage-level cost analysis also helps explain the sources of the savings.

The authors provide results across three repositories and three agent configurations, along with NDI ablations, per-repository results, and confidence intervals for some comparisons, allowing readers to examine the scope of the conclusions.

Critical weaknesses
  1. The paper validates task selection according to declarations, but does not establish the effectiveness of this customization as an evaluation methodology.

The introduction motivates the work through adaptation to team workloads, diagnosis of specific capabilities, and difficulty control. However, “tasks satisfy user-specified properties” does not automatically imply that “these tasks represent the user’s work or accurately measure the target capability.” For a one-off task, consulting public evaluations and directly trying an agent is already a reasonable approach. The additional value of a customized benchmark should primarily arise in repeatable selection across many tasks or capability diagnosis, but the paper does not concretely demonstrate this value.

This issue also appears in the main example, NDI: the authors interpret it as navigation difficulty, but fit it using overall task outcomes, while factors such as file count, directory count, and import-graph distance receive zero weight in Equation (6). Its aggregate AUC and that of the LLM rating alone both round to 0.67. Thus, the results show that it can select harder tasks, but do not establish that it isolates navigation capability or provides more effective customization than an existing difficulty rating.

I am not asking for a large-scale user study, but at least one case with a clearly defined objective is needed to establish the correspondence between the declarations and the target workload or capability, and to validate the resulting benchmark’s relevance to that objective. Otherwise, the demonstrated contribution is closer to a configurable task filtering and construction tool than a validated methodology for customized evaluation.

  1. The optimizer’s technical contribution is unclear, and the current description does not distinguish a new method from the application of existing execution optimization principles.

Section 2.3 proposes jointly optimizing filter order and measurement batching, but primarily describes principles such as evaluating inexpensive conditions early, prioritizing highly selective conditions, and sharing computation. These are closely related to existing work on declarative data processing and query optimization. The paper does not specify the algorithms for cost estimation or joint ordering and batching, nor does it explain how the dependency constraints introduced by Transform are handled during planning.

This is more than a missing implementation detail: the optimizer is a central technical claim, yet the manuscript does not provide enough information to assess its new mechanisms. The paper needs to specify the algorithm and explain which task-construction problems cannot be addressed directly through simple early filtering, shared computation, or existing planning methods.

  1. The main efficiency experiments do not clearly separate changes in selection requirements from execution optimization, and therefore do not directly substantiate the claimed optimization gains.

Section 3.1 states that the baseline only filters for changes exceeding 50 source lines, whereas swespoke additionally filters on the number of changed files, change locations, and test assertions. The paper does not clarify whether the baseline ultimately applies the same conditions and produces the same task set. If the two pipelines have different final requirements, the cost reduction from processing fewer candidates cannot be interpreted as an execution speedup for satisfying the same declaration.

Similarly, the measurement experiment uses a baseline that computes features independently, and Table 2 shows that the gains primarily come from avoiding repeated parsing and statistics computation. This supports the engineering value of shared computation, but does not establish the joint optimizer’s advantage over a reasonable implementation with caching and heuristic execution.

A comparison using the same inputs, declarations, and final task requirements is therefore needed, together with a baseline using simple early filtering and shared caching. This comparison directly determines how the paper’s main efficiency gains should be interpreted.

Questions for authors

Can you provide a concrete case validating that the declared properties correspond to the target workload or capability, rather than merely changing the task subset? What evidence supports interpreting NDI specifically as navigation difficulty?

What are the joint optimizer’s concrete algorithm and new mechanisms? What additional problem does it solve beyond simple early filtering and shared computation?

Do the two pipelines in Figure 3 and Table 4 ultimately satisfy the same selection requirements? How much of the reported gain remains when the conditions are matched?

2 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer WRZV (Part 1 of 2)

Thank you for your constructive feedback. We apologize for the delay in our response. Our lead author had a medical emergency, so we were unable to respond earlier in the rebuttal period. We hope you will still consider our responses and the revised paper. We would be happy to answer any follow-up questions.

W1: On workload relevance: we agree that satisfying a declaration alone does not establish that the resulting tasks are useful for a particular evaluation. One important use case of swespoke is to help users evaluate and improve coding agents on work drawn from their own repositories. To demonstrate this, we added a RocksDB case study in Section 3.4, focused on improving Claude Code’s performance on changes that span multiple source files. We used swespoke to create tasks from RocksDB pull requests that modified at least two source files. We split 100 such tasks into 50 training, 20 validation, and 30 test tasks with similar TDI distributions. We then used GEPA to refine Claude Code’s repository instructions (CLAUDE.md) from its execution traces and test results. On the 30 RocksDB test tasks, Claude Code’s pass rate rose from 66.7% (20/30) without the added repository instructions to 76.7% (23/30) with the revised CLAUDE.md, a gain of 10 percentage points. This case shows how users can use failures on tasks from their own repository to evaluate and refine their coding agents. As new pull requests arrive, users can periodically rerun their customized swespoke declaration to select new tasks and update a customized CLAUDE.md for each repository.

On the interpretation of the difficulty index: we agree that fitting NDI to overall task outcomes provides insufficient evidence that it measures navigation difficulty. One possible explanation for the low weights on navigation-related factors is that our sample of tasks from three repositories captures limited variation in those factors. To align the name with our current results, we renamed NDI the Task Difficulty Index (TDI) and revised the introduction and Section 2.5 to focus on selecting failure-prone tasks under a fixed evaluation budget. In our earlier evaluation, TDI had a slightly higher observed aggregate AUC than the standalone LLM rating (0.673 versus 0.669). Customization also lets users change TDI’s inputs or refit its weights as their agents evolve.

W2: We defined the optimizer’s objective in Section 2.1, presented its high-level workflow in Section 2.4, and added pseudocode for staged task creation (Algorithm 1) and filter execution planning (Algorithm 2) in Appendix A. Specifically, we design a greedy algorithm to reduce the cost of filter checks and the task creation work they require, since even finding the cheapest filter order is NP-hard. Within each stage, the planner prioritizes eligible filters by estimated pruning rate per dollar while respecting measurement dependencies and transformation order. It estimates costs from direct charges and server rental and updates the plan every h candidates using observed costs and pruning rates. It batches compatible measurements when computing them together is expected to cost less than running them separately.

The main technical challenge is that task selection and task creation depend on each other. Without a general way to integrate selection into task creation, users must manually decide where to check each new requirement and how to act on the result. Our contribution is to formalize this problem and design the abstraction of the filters that enable users to specify their selection requirements without modifying the task construction pipeline. None of the task construction approaches we examined, including SWE-bench-Live, SWE-Factory, and Repo2RLEnv, describes a cost-based optimizer for automatically applying user-defined filters during task creation. We also agree that early pruning and batching are established optimization principles. The novelty lies in modeling how pruning and batching affect the cost of creating the requested task set and using those estimates to decide when to apply each filter and which measurements to compute together.

AuthorsOpenReview ↗

Authors’ Response to Reviewer WRZV (Part 2 of 2)

W3: For task creation: both the baseline and swespoke start from the same pull requests, apply the same final selection criteria, and return the same tasks in each repository. Both first require more than 50 changed source lines. The only difference is that swespoke automatically applies the file-count, change-location, and assertion-count filters, as well as the description–test alignment check, to prune task candidates during construction, whereas the baseline applies those three filters and excludes flagged tasks after construction. We clarified the setup in Section 3.1 and the results in Figure 3 and Table 2.

For measurement collection: we agree that most of the speedup comes from reusing parsing results and intermediate statistics, as reported in Table 3. swespoke manages this reuse automatically, so users do not have to write and maintain their own code to cache intermediate results. We also added a baseline that computes each shared intermediate once per task, saves it to disk, and reloads it when needed. swespoke collected the 63 measurements 1.93–2.58× as fast as this cached baseline by grouping measurements and reusing intermediates in memory (Appendix C, Table 3).

Q1: As discussed in W1, our RocksDB case study targets changes that span multiple source files. We used swespoke to create tasks from RocksDB’s own pull requests and required each selected change to modify at least two source files. We then used GEPA to refine Claude Code’s CLAUDE.md from its execution traces and test results. On the same 30 RocksDB test tasks, Claude Code passed 23 with the revised CLAUDE.md, compared with 20 without the added instructions (Section 3.4). This comparison shows that feedback from the selected tasks helped improve Claude Code’s performance on the RocksDB workload specified by the declaration.

We agree that the evidence in the current experiments is insufficient to interpret NDI as a measure of navigation difficulty. We therefore renamed it the Task Difficulty Index (TDI) and revised the introduction and Section 2.5 to focus on selecting failure-prone tasks under a fixed evaluation budget.

Q2: As discussed in W2, we added the optimization objective in Section 2.1, revised Section 2.4 to explain the planning algorithm, and added pseudocode in Appendix A. swespoke uses a greedy algorithm that orders eligible filters within each stage by estimated pruning rate per dollar and then decides which measurements to batch, while respecting measurement dependencies and the required order of transformations.

Our contribution is to formalize this planning problem and design a filter abstraction that enables users to specify their selection requirements without manually deciding where to apply each filter during task creation. We agree that early pruning and batching are established optimization principles. The novelty lies in modeling how they affect the cost of creating the requested task set and using those estimates to decide when to apply each filter and which measurements to compute together.

Q3: Yes. Both pipelines apply the same final selection criteria and return the same 202 RocksDB tasks, 341 Polars tasks, and 488 DuckDB tasks (Table 2). With these matched criteria, swespoke achieves 7.27–12.48× faster task creation and 2.39–4.85× lower LLM expense. As discussed in W3, the only difference is that swespoke automatically applies filters and prunes candidates during construction, while the baseline applies the non-built-in filters after construction. Most of the savings come from skipping LLM assessment, environment creation and verification for early rejected candidates. We clarified this comparison in Sections 3.1 and 3.2.

Reviewer_9RAAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces swespoke, a framework for constructing software-engineering benchmarks from user-declared task properties. A measurement–condition–action interface specifies task selection, reporting, and revision, while dependency-aware execution and shared computation aim to reduce unnecessary environment construction and testing. The paper also proposes NDI, a difficulty index combining change characteristics, task descriptions, and LLM difficulty ratings, with weights fitted to agent outcomes. Experiments cover RocksDB, Polars, and DuckDB, producing 223 execution-verified tasks from 31,677 candidate PRs and evaluating three coding agents. The paper reports 7.27–12.48× faster task creation, 2.39–4.85× lower LLM expense, and 14.43–19.94× faster measurement collection. NDI improves failure-prediction AUC over changed LOC by approximately 0.07, although its rounded aggregate AUC matches the standalone LLM rating at 0.67.

Critical strengths
  1. The measurement–condition–action interface separates user requirements from execution strategy while accounting for measurement dependencies and recomputation after task revisions. This is a coherent systems design for constructing benchmarks tailored to different users, and the paper’s strongest contribution.
  2. The system generates tasks from three substantial C++/Rust repositories, identifies P2P/F2P tests through execution before and after reference changes, and evaluates all 223 tasks with three agents. This supports practical feasibility, while Table 2 provides concrete evidence of the benefits of shared computation.
Critical weaknesses
  1. In Section 3.1, the filtering baseline requires only more than 50 changed source lines, whereas swespoke applies additional file-count, change-location, and assertion-count conditions. It is unclear whether the baseline eventually applies identical criteria and produces the same task set. Consequently, the creation-time reduction may combine changed selection criteria with execution optimization. For measurement collection, the baseline independently recomputes features, and Table 2 attributes most gains to shared parsing and statistics. Together with the missing planning algorithm in Section 2.3, these results establish the benefits of pruning and reuse more directly than the distinct contribution of the proposed joint optimizer.
  2. Section 3.3 reports fitting NDI on 468 of 669 runs without specifying whether splitting occurs by task or individual agent run. Because each task has outcomes from three agents, this distinction is essential for assessing generalization. The aggregation of held-out results and the bootstrap resampling unit are also unspecified. Furthermore, most structural and navigation factors receive zero weight in the fitted formula, the search term contributes little, and aggregate AUC matches the standalone LLM rating. The results support selecting failure-prone tasks, but do not yet establish navigation-specific difficulty or a reliable advantage over the simpler rating.
  3. The semantic validity of the generated tasks is insufficiently validated. Although the system performs F2P/P2P checks and uses an LLM to screen interface mismatches, these checks do not establish that verifiers accept reasonable correct implementations, reject incorrect ones, and avoid unstated requirements. The paper reports neither an independent task-quality audit nor the alignment checker’s accuracy. Consequently, the evidence supports the feasibility of automated construction, but does not adequately establish the resulting benchmarks’ reliability for measuring agent capability. This also complicates interpreting higher failure rates as greater task difficulty.
Questions for authors
  1. What are the paired differences and confidence intervals between NDI and the standalone LLM rating for AUC and hardest-k% selection? Given the limited contribution of navigation terms in the fitted formula, what analysis directly supports its interpretation as navigation difficulty?
  2. Beyond execution-based validation and the LLM-based interface-mismatch check, what independent evidence do the authors provide that the retained task descriptions and their verifiers are semantically aligned?
2 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer 9RAA (Part 1 of 2)

Thank you for your constructive feedback. We apologize for the delay in our response. Our lead author had a medical emergency, so we were unable to respond earlier in the rebuttal period. We hope you will still consider our responses and the revised paper. We would be happy to answer any follow-up questions.

W1: For task construction: both pipelines apply the same final selection criteria and produce the same task set. swespoke applies the file-count, change-location, and assertion-count filters during construction and prunes candidates flagged by the description-test alignment check. The baseline applies the three filters and excludes tasks with a description-test mismatch after construction. Its reported cost includes this final filtering. For example, on RocksDB, the baseline sends 2,055 candidates to verification and swespoke sends 202, yet both return the same final 202 tasks. We clarified the setup in Section 3.1, and updated Figure 3 and Table 2 to make this distinction explicit.

For measurement collection: we agree and have also reported that most of the speedup in Table 3 comes from sharing parsing and intermediate statistics. swespoke manages this reuse automatically, so users do not have to write and maintain their own code to cache intermediate results. We added a baseline that computes each shared intermediate once per task, saves it to disk, and reloads it when needed. swespoke collected the 63 measurements 1.93–2.58× as fast as this cached baseline by grouping measurements and reusing intermediates in memory (Appendix C, Table 3).

For the planning algorithm: we added a new Section 2.1 to explain the planning targets, updated Section 2.4 to describe the planning algorithm, and added Algorithms 1 and 2 in Appendix A to explain the task creation and plan optimization algorithms in detail. Because the planning problem is NP-hard, we design a greedy algorithm. The planner orders eligible filters by estimated pruning rate per dollar after resolving the dependencies, then uses that order to decide whether computing measurements together costs less than computing them separately. The experiments quantify the savings from early pruning and shared computation.

W2: We updated Sections 3.2 and 3.3 to clarify the training and test split, and added evaluation details to Appendix D to explain aggregation and bootstrapping. Specifically, (1) we split the 223 tasks within each repository into one training set of 156 tasks and one test set of 67 tasks, keeping all three agents’ runs on a task together. We fit one TDI formula using the 468 training runs and evaluate it on the 201 test runs. (2) We compute AUC separately for each agent and repository, weight by the number of test tasks, and average across agents. We select the hardest k% of test tasks within each repository and pool the selected runs to calculate the failure rate. All point estimates use the original test set. For the 95% confidence intervals, we draw 1,000 bootstrap samples with replacement within each repository, preserving its test-set size and keeping each task’s three outcomes together. Each sample contains 67 task draws.

We agree that, overall, the fitted NDI formula provides insufficient evidence for interpreting it as a measure of navigation difficulty. One possible explanation is that evaluating only three projects limits the variation in navigation-relevant factors. To make the name consistent with what the index measures on the current results, we renamed NDI to Task Difficulty Index (TDI) and revised the introduction and Section 2.5 to focus on selecting failure-prone tasks under a fixed evaluation budget. TDI provides a continuous score where the standalone LLM rating has three levels, and both TDI’s observed aggregate AUC (0.673 against 0.669) and hardest-k% task failure rates (53.0% against 50.5% at 10%, and 52.7% against 50.9% at 20%) are higher than those of the LLM rating.

AuthorsOpenReview ↗

- Authors’ Response to Reviewer 9RAA (Part 2 of 2)

W3: We would first like to clarify that swespoke can use any automated task-quality check as a measurement, so users can easily add or replace a checker without having to manage how it is applied during task selection and creation. Therefore, designing a reliable quality checker is orthogonal to the task selection problem that swespoke addresses. In future work, we will systematically study how to assess task quality.

That said, we agree that it is important to validate the quality of the tasks used in our experiments. We added two new task-quality assessments in Section 3.3 and Appendix E. First, we compared swespoke’s task description-test alignment checker and the SWE-bench-Live prompt against independent human ratings on the same 1,699 tasks reviewed for SWE-bench Verified (Chowdhury et al., 2024). swespoke retained 455 tasks as aligned, of which 324 (71.2%) had tests that human reviewers rated acceptable. The corresponding share for the SWE-bench-Live prompt was 553 of 1,027 (53.8%). Among retained tasks, 10.8% received the most severe rating with swespoke, compared with 18.2% with SWE-bench-Live (Table 9).
Second, on 210 of the 223 swespoke tasks (94.2%), at least one of six coding-agent configurations produced a submission that the grader accepted (Table 8). Solvability matters because flawed tests can reject correct solutions, as OpenAI found in its audit of 138 SWE-bench Verified tasks that o3 struggled to solve consistently (OpenAI, 2026). Together, these two assessments show that a larger share of the tasks retained by swespoke met the human reviewers’ standard for test fairness and that most swespoke tasks in our experiments were solvable.

Q1: We compare TDI and the standalone LLM rating on the same 67 held-out tasks (Appendix D, Table 4). The paired difference in aggregate AUC (TDI minus the LLM rating) is +0.004 (95% CI −0.025 to +0.037). At the 10% and 20% budgets, the tasks selected by TDI have observed failure rates 2.5 percentage points higher (95% CI −5.7 to +8.2 points) and 1.8 points higher (95% CI −2.6 to +5.9 points), respectively.

As discussed in W2, we agree that overall task outcomes alone provide insufficient evidence that the index measures navigation difficulty. To align the name with our current results, we renamed it the Task Difficulty Index and revised our claim to focus on selecting failure-prone tasks under a fixed evaluation budget.

Q2: As discussed in W3, we added two studies to assess task quality (Section 3.3 and Appendix E). First, swespoke’s description-test alignment checker retained 455 of the 1,699 tasks reviewed for SWE-bench Verified. According to the published annotations, 324 of the tasks that swespoke marked as aligned (71.2%) had acceptable tests, compared with 553 of the 1,027 tasks (53.8%) retained by the SWE-bench-Live prompt. Second, on 210 of the 223 swespoke-generated tasks (94.2%), at least one of six coding-agent configurations produced a submission that the grader accepted. Together, these two studies provide complementary evidence of task quality.

We agree that assessing task quality is very important. Because swespoke works with any automated quality checker, checker design is orthogonal to the task selection problem that swespoke addresses. We updated the conclusion to include a systematic study of methods for assessing task quality as future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Major Revisions

We thank the reviewers for their constructive feedback. We apologize for the delay in our response. Our lead author had a medical emergency, so we were unable to respond earlier in the rebuttal period. We hope the area chair and reviewers will still consider our responses and the revised paper. We would be happy to answer any follow-up questions.

In summary, we made the following five major revisions to the paper to address the reviewers’ concerns and suggestions. We address each comment point by point in our individual replies.

  1. We formalized the task selection problem and described the optimizer. Section 2.1 defines the objective of minimizing the cost of filter checks and the task creation work they require. Section 2.4 explains how swespoke orders eligible filters by estimated pruning rate per dollar and decides when to batch measurements. Appendix A gives pseudocode for generating and executing the plan.

  2. We clarified the task construction comparison and added a cached measurement baseline. Both pipelines apply the same final selection criteria. The baseline applies the additional filters after construction, while swespoke automatically applies them during construction to prune candidates earlier. For measurement collection, swespoke computes all 63 measurements 1.93–2.58× as fast as the new baseline, which caches shared intermediate results on disk (Sections 3.1–3.3 and Appendices B–C).

  3. We revised the interpretation and evaluation of the difficulty index. We renamed NDI to Task Difficulty Index (TDI), and repurposed it to help users with a fixed evaluation budget select tasks on which their coding agents are more likely to fail. We also clarified how tasks are split between training and testing, how results are aggregated, and how paired bootstrap intervals are calculated for comparisons with the standalone LLM rating (Sections 2.5 and 3.2–3.3 and Appendix D).

  4. We added two assessments of task quality. According to the published human annotations for the 1,699 tasks reviewed for SWE-bench Verified, 71.2% of the tasks retained by swespoke’s alignment checker had acceptable tests, compared with 53.8% of those retained by SWE-bench-Live. On swespoke’s own tasks, at least one of six coding-agent configurations produced a passing submission for 210 of 223 tasks (94.2%), providing evidence that most were solvable (Section 3.3 and Appendix E).

  5. We added a RocksDB case study showing how a team can use customized tasks to improve its coding agent. We selected tasks from RocksDB pull requests that changed multiple source files and used GEPA to refine Claude Code’s repository instructions (CLAUDE.md) from its execution traces and test results. On the same 30 test tasks, Claude Code passed 23 with the revised instructions and 20 without them, a gain of 10 percentage points (Section 3.4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.