- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TestSuiteBench, a benchmark for evaluating whether coding agents can write complete test suites from natural-language specifications, without access to the implementation. Existing benchmarks often focus on individual functions or bugs and provide the code to be tested. TestSuiteBench instead covers 165 open-source applications, with a detailed specification, a reference implementation and test suite, and an average of 74 mutants per application. Each mutant is designed to violate one requirement in the specification. The authors refine the mutants so that 73% are killed by exactly one reference test, and each is killed by only about 0.2% of the other reference tests on average. The results show that even frontier agents struggle to generate strong test suites: the best mutation score is 55.5%, and four of the six models score below 40%. On average, 88% of the generated tests pass on the reference implementation, and implementations generated from the same specifications pass about 82% of the reference tests. The generated test suites tend to be smaller than the reference suites and often miss relevant input combinations.
Critical strengths
- The paper addresses an important gap: whether coding agents can create a complete test suite from a specification, rather than adding tests for a known bug or testing a provided implementation. TestSuiteBench offers a way to measure this capability at the repository level.
- The benchmark uses requirement-targeted mutants to measure test strength. The authors also filter reference tests that rely on network access, exceed resource limits, or are flaky. They refine the mutants so that 73% are killed by exactly one reference test. This gives a more focused measure than using generic mutations, although the representativeness of the mutants remains an open question.
- The paper measures both test generation and implementation generation. The generated tests are often valid, and implementations built from the same specifications often pass the reference tests, while mutation scores remain low. This supports the paper’s main finding that agents can build software they cannot adequately test.
- The authors provide a pipeline for creating and refining specifications and mutants, and they manually review the resulting artifacts. This makes it possible to build a benchmark across many repositories, though extending it to other languages and application types will require further work.
Critical weaknesses
- The specifications are much more detailed than the documentation commonly available in open-source projects. Their median length is 5,635 words across 23 sections, and they are derived from reference tests and refined through agent trials. This makes the benchmark useful for measuring test generation given a clear specification, but less representative of settings where agents must work from brief or incomplete requirements. The paper compares its specifications with developer documentation, but the main evaluation still relies on the more detailed specifications.
- The authors aim for mutants that are killed by only one reference test. They also audit mutants to reduce the risk of hard-coding behavior for specific inputs, and they discuss the remaining limitations. Even so, this design may favor a narrow set of faults or behaviors. It would be useful to know how agent rankings and mutation scores change when mutants have broader kill sets or reflect a wider range of fault types.
- The benchmark is based on PROGRAMBENCH and currently covers command-line applications written mainly in Rust, Go, and C/C++. It does not evaluate applications in interpreted languages or other settings such as web services. This limits how far the results can be generalized, although the authors note that the pipeline could be extended to languages such as Python and TypeScript.
- The paper shows examples where tests miss combinations of behaviors, but it does not provide a systematic analysis of why this happens. For example, it is unclear whether agents miss these cases because of limited test-generation strategies, the size of the specifications, or the prompts and tools they use. More analysis could help identify ways to improve test generation.
Questions for authors
- What changes would be needed to extend the TestSuiteBench construction pipeline to GUI applications or web APIs, where behavior may depend on sequences of actions or state changes?
- The paper aims to create mutants that are killed by a small number of reference tests, and reports that 73% are killed by exactly one. The authors also audit mutants to avoid ones that depend on specific hard-coded inputs. Could you test how the results change with mutants that have broader kill sets or represent a wider range of real-world faults? This would help clarify how much the benchmark’s scores depend on the singleton-kill-set design.
- The examples show that generated tests can cover individual requirements but miss combinations of behaviors. Have you tested whether different prompts, test-generation strategies, or agent tools help address this problem?
We thank the reviewer for the thorough assessment of our work. Below, we address each of the points the reviewer raised.
W1: Is it realistic to use the detailed specifications as input?
Yes, please refer to the global answer (G2), where we highlight (i) the practical implications of a large-spec-based development, and (ii) that we carefully design it to make it more natural than prior benchmarks.
W2 & Q2: Is the mutation score representative of a wider range of real-world faults?
Yes, please refer to the global response (G1), where we provided a new experimental result showing a strong correlation between our mutation score and real-bug detectability.
W3 & Q1: Elaborate on how it could be extended to other domains.
The high-level idea of our benchmark construction is to prepare a consistent set of specifications, mutants, and a reference implementation and test suite. In particular, specifications, mutants, and reference implementations should capture the same input-output behavior. This pattern can be applied beyond CLI programs to domains such as GUI applications and web APIs: Instead of defining inputs in terms of command-line flags and arguments, inputs can be (a sequence of) click or drag operations in GUI applications and arguments to API endpoints for web APIs. For outputs, instead of stdout and stderr outputs, the tests and specification can state assumptions about visible elements in GUI applications and returned values or observable state changes in web APIs. We include a discussion of such extensions in our discussion and limitations section (See Section 5).
W4: Provide more systematic analysis of failure cases.
Yes, please refer to the global answer (G3). We analyzed the surviving mutants and found that the generated test suites often exercise relevant code, but fail to trigger mutated behavior, likely due to insufficient input diversity. When differences do surface, tests often detect them.
Q3 & W4: Explore different test generation approaches.
We provide the following new results. If there are some other specific experiments that the reviewer is interested in, we are happy to evaluate them.
Explicitly instructing to create executable/mutants. As we found in Table 3, Opus (the best-performing model) often creates executables and mutants to check the test. We additionally investigated whether an explicit instruction to create them could improve the mutation score for other models. As shown in the new entry of Table 3 (also pasted in this reply), we found that this strategy slightly decreased the mutation score (37% -> 34%) for Sol. We observe that the total time/cost remains the same, which implies the model spent less effort generating the test suite itself; in fact, the number of generated test functions substantially decreased (on average 125 -> 98). Therefore, executable/mutation-based test generation is not a universal solution, and a deeper look into its influence is left as an interesting avenue for future work.
Table: the new result, “Sol + prompt”, explicitly asks the model to create executables and mutants to check its test suite.
Multi-turn evaluation. In practice, specifications may evolve, requiring iterative test updates. We simulate such a use case by revealing the specification in five increments, asking the agent to update its previous test suite each time. As shown in the new Table 7, we found that this improves mutation score ($\Delta=9.2pp),particularlycontributingforlargespecifications,weakeningthenegativecorrelationbetweenspecsizeandmutationscore(\rho=-0.42$ to −0.27). However, cost increases by 3.5×, time by 1.4×, and validity drops by 6.3%. Importantly, xhigh reasoning effort of the same model achieves a comparable mutation score (46.5%) at a lower cost ($2.3) in a single-turn setup, suggesting that reasoning effort solves the cost-quality tradeoff better than running the model many times with a focused set of requirements.
Table: Five iterations with Sol at medium reasoning effort. Changes are relative to the main setup.