- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
-
The paper introduces a framework for creating tailored SWE-bench-style tasks from existing repository histories. It lets users declare measurements, conditions, and actions to select or revise tasks based on desired properties and difficulty. The framework optimizes filter order and shares computations to avoid unnecessary construction work
-
Authors run experiments on RocksDB, Polars, and DuckDB to produce 223 tasks, reporting 7-12x faster construction and 2-4× lower LLM expense than a simpler baseline pipeline. The authors also propose a navigation difficulty index (NDI) for selecting harder tasks as an example usage of the framework, reducing average agent pass rate from 68% overall to 46% on the hardest 10%
Critical strengths
-
The paper addresses a practical issue of task construction efficiency for SWE Agent tasks. Applying pre-filtering is a very reasonable practices that practitioners adopt to produce desirable tasks.
-
The measurement-condition-action framework provides a flexible way to tailor SWE Agent tasks construction pipelines without big refactoring for different requirements.
Critical weaknesses
-
Early filtering and shared computation are practical ways to reduce task-construction costs, but the paper does not clearly establish methodological novelty beyond established filter ordering, batching, and computation reuse. The contribution appears primarily to be an engineering framework
-
The reported cost reductions are not accomplished by proper comparison of task quality, such as task fairness, test coverage, the noise ceilings of the benchmark, etc. It is also unclear whether the baseline applies identical final selection criteria. Without matched output requirements or independent quality assessment, the efficiency results are difficult to interpret
-
The framework focuses on single PR, SWE-bench styled tasks, while the industry have moved on to more broader task ranges and diverse classifiers (e.g. DeepSWE, SWE Atlas). The framework feels more relevant to the construction of training data instead of benchmarking, yet quality signal on the produced tasks as training data is missing
Questions for authors
-
Does the simpler baseline apply the same selection criteria as swespoke after task construction, including the multiple-source-file requirement, or does it retain tasks that swespoke rejects?
-
Could the authors assess the quality of swespoke-generated tasks, and compare against an established task-construction method such as SWE-smith?
2 comments
Authors’ Response to Reviewer 68qz (Part 2 of 2)
W3: On single-PR tasks: we agree that swespoke currently creates each task from a single PR. We appreciate the reviewer’s observation that software engineering evaluations are expanding to cover a broader range of work. We updated our future work discussion to include these long-horizon and more diverse engineering tasks, alongside our existing plan to automate multi-PR task creation. However, we also observe that single PRs still offer a practical way for engineering teams to create evaluation tasks from their own repositories, as shown by recent work at Databricks, Red Hat, and ServiceNow (Gaba et al., 2026; Bende, 2026; Hinckley et al., 2026). Also, as our evaluation shows, current coding agents still fail on these specialized tasks, giving engineering teams feedback they can use to improve their agents.
On the utility of swespoke tasks: we added a RocksDB case study in Section 3.4 to illustrate how swespoke can help users evaluate and improve coding agents on work drawn from their own repositories. We tested whether feedback from RocksDB tasks could improve Claude Code’s performance on changes that span multiple source files. We used swespoke to create tasks from RocksDB pull requests that modified at least two source files. We split 100 such tasks into 50 training, 20 validation, and 30 test tasks with similar TDI distributions. We then used GEPA to refine Claude Code’s repository instructions (CLAUDE.md) from its execution traces and test results. On the 30 RocksDB test tasks, Claude Code’s pass rate rose from 66.7% (20/30) without the added repository instructions to 76.7% (23/30) with the revised CLAUDE.md, a gain of 10 percentage points.
This case shows how users can use failures on tasks from their own repository to evaluate and refine their coding agents. As new pull requests arrive, users can periodically rerun their customized swespoke declaration to select new tasks and update a customized CLAUDE.md for each repository.
References:
Vinay Gaba, Ankit Mathur, Rishabh Singh, Patrick Wendell, and Matei Zaharia. July 8, 2026. Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase. Databricks Blog.
Rounak Bende. October 21, 2026. Red Hat enterprise-specific benchmarks for coding agents: Why we built RH SWE-bench. Red Hat Emerging Technologies.
Chris Hinckley, Andrea Tan, Sherry Han, Estelle Hu, Meiru Han, Zianyi Wang, Om Babar, Erin Lafrenz, and Aaliyah Brown. November 4, 2026. Benchmarking Coding Agents on the Now Platform Codebase. ServiceNow Developer Blog.
Q1: Yes. Both pipelines apply the same final selection criteria and return the same 202 RocksDB tasks, 341 Polars tasks, and 488 DuckDB tasks (Table 2). As discussed in W2, the only difference is that swespoke automatically applies filters and prunes candidates during construction, while the baseline applies the non-built-in filters after construction. Most of the savings come from skipping LLM assessment, environment creation and verification for candidates rejected early. We clarified this comparison in Sections 3.1 and 3.2.
Q2: As discussed in W2, we added two task-quality assessments in Section 3.3 and Appendix E. First, we compared swespoke’s task description-test alignment checker with the SWE-bench-Live prompt on the same 1,699 tasks reviewed for SWE-bench Verified. According to the published human annotations, 71.2% of the tasks retained by swespoke had acceptable tests (324/455), compared with 53.8% of those retained by the SWE-bench-Live prompt (553/1,027) (Table 9). Second, on 210 of the 223 swespoke tasks (94.2%), at least one of six coding-agent configurations produced a submission that the grader accepted (Table 8). Together, these assessments support using swespoke’s checker to select tasks with fair tests and show that most of the generated tasks were solvable.
SWE-smith creates a different type of task by introducing bugs that cause existing tests to fail, making a direct comparison with swespoke’s PR-based tasks difficult. We instead chose SWE-bench-Live as an established task-construction pipeline and compared its prompt and swespoke’s checker against published human judgments on the same types of tasks. We agree that assessing task quality is important. Because swespoke works with any automated quality checker, checker design is orthogonal to the task selection problem that swespoke addresses. We updated the conclusion to include a systematic study of methods for assessing task quality as future work.
Authors’ Response to Reviewer 68qz (Part 1 of 2)
Thank you for your constructive feedback. We apologize for the delay in our response. Our lead author had a medical emergency, so we were unable to respond earlier in the rebuttal period. We hope you will still consider our responses and the revised paper. We would be happy to answer any follow-up questions.
W1: The technical problem that swespoke resolves is how to automatically apply a user’s selection requirements during task creation while minimizing the cost of producing the requested task set. This is challenging because task selection and task creation depend on each other. For example, some requirements can be checked directly from repository history, while others require building an environment and running tests. Without a general way to integrate selection into task creation, users have to manually decide where to check each new requirement and how to act on the result.
Our contribution is to formalize task selection during task creation as a constrained cost-minimization problem and to design the abstraction of filters that enables users to specify any selection requirements without manually modifying the construction pipeline. None of the task construction approaches we examined, including SWE-bench-Live, SWE-Factory, and Repo2RLEnv, describes a cost-based optimizer for automatically applying user-defined selection criteria during task creation. We also agree that early pruning and batching are established optimization principles. The novelty lies in modeling how pruning and batching affect the cost of creating the requested task set and using those estimates to decide when to apply each filter and which measurements to compute together.
We clarified these contributions in Sections 2.1 and 2.4 of the revised paper.
W2: For task selection criteria: we would like to clarify that both swespoke and the baseline apply the same final selection criteria and return the same 202 RocksDB tasks, 341 Polars tasks, and 488 DuckDB tasks (Table 2). The only difference is that swespoke automatically applies the file-count, change-location, and assertion-count filters, as well as the description–test alignment check, to prune candidates during construction, whereas the baseline applies those three filters and excludes flagged tasks after construction. The savings mainly come from skipping LLM assessments, environment creation, and verification for candidates rejected early. We clarified the setup in Section 3.1 and the results in Figure 3 and Table 2.
For task quality: we would also like to clarify that swespoke can use any automated task-quality check as a measurement, so users can add or replace a checker without having to manage how it is applied during task selection and creation. Designing a reliable quality checker is therefore orthogonal to the task selection problem that swespoke addresses. In future work, we will systematically study how to assess task quality, including task fairness, test coverage, and benchmark noise ceilings.
That said, we agree that it is important to validate the quality of the tasks used in our experiments. We added two task-quality assessments in Section 3.3 and Appendix E. First, we compared swespoke’s description–test alignment checker and the SWE-bench-Live prompt against independent human ratings on the same 1,699 tasks reviewed for SWE-bench Verified (Chowdhury et al., 2024). swespoke marked 455 tasks as aligned, of which 324 (71.2%) had tests rated acceptable in the published annotations. The corresponding share for the SWE-bench-Live prompt was 553 of 1,027 (53.8%). Among the tasks retained by each checker, 10.8% received the most severe rating for swespoke, compared with 18.2% for SWE-bench-Live (Table 9).
Second, on 210 of the 223 swespoke tasks (94.2%), at least one of six coding-agent configurations produced a submission that the grader accepted (Table 8). Solvability matters because flawed tests can reject correct solutions, as OpenAI found in its audit of 138 SWE-bench Verified tasks that o3 struggled to solve consistently (OpenAI, 2026). Together, these assessments show that a larger share of the tasks retained by swespoke met the human reviewers’ standard for test fairness and that most swespoke tasks in our experiments were solvable.