acceptodds
Under review as a conference paper at ICLR 2027

TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications

Abstract

Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes. In particular, generating complete test suites for complex software from scratch remains unexplored. To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73% are killed by exactly one reference test and, on average, by only 0.2% of the remaining tests. We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60%, despite their suites being largely valid and 88% of their tests passing on the reference implementatio. We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. Meanwhile, implementations generated for the same specifications pass 82% of the reference tests. These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.

Primary area:
datasets and benchmarks
Keywords:
LLM, coding agent, test suite generation, mutation testing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_J7tzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TestSuiteBench, a benchmark for evaluating whether coding agents can write complete test suites from natural-language specifications, without access to the implementation. Existing benchmarks often focus on individual functions or bugs and provide the code to be tested. TestSuiteBench instead covers 165 open-source applications, with a detailed specification, a reference implementation and test suite, and an average of 74 mutants per application. Each mutant is designed to violate one requirement in the specification. The authors refine the mutants so that 73% are killed by exactly one reference test, and each is killed by only about 0.2% of the other reference tests on average. The results show that even frontier agents struggle to generate strong test suites: the best mutation score is 55.5%, and four of the six models score below 40%. On average, 88% of the generated tests pass on the reference implementation, and implementations generated from the same specifications pass about 82% of the reference tests. The generated test suites tend to be smaller than the reference suites and often miss relevant input combinations.

Critical strengths
  1. The paper addresses an important gap: whether coding agents can create a complete test suite from a specification, rather than adding tests for a known bug or testing a provided implementation. TestSuiteBench offers a way to measure this capability at the repository level.
  2. The benchmark uses requirement-targeted mutants to measure test strength. The authors also filter reference tests that rely on network access, exceed resource limits, or are flaky. They refine the mutants so that 73% are killed by exactly one reference test. This gives a more focused measure than using generic mutations, although the representativeness of the mutants remains an open question.
  3. The paper measures both test generation and implementation generation. The generated tests are often valid, and implementations built from the same specifications often pass the reference tests, while mutation scores remain low. This supports the paper’s main finding that agents can build software they cannot adequately test.
  4. The authors provide a pipeline for creating and refining specifications and mutants, and they manually review the resulting artifacts. This makes it possible to build a benchmark across many repositories, though extending it to other languages and application types will require further work.
Critical weaknesses
  1. The specifications are much more detailed than the documentation commonly available in open-source projects. Their median length is 5,635 words across 23 sections, and they are derived from reference tests and refined through agent trials. This makes the benchmark useful for measuring test generation given a clear specification, but less representative of settings where agents must work from brief or incomplete requirements. The paper compares its specifications with developer documentation, but the main evaluation still relies on the more detailed specifications.
  2. The authors aim for mutants that are killed by only one reference test. They also audit mutants to reduce the risk of hard-coding behavior for specific inputs, and they discuss the remaining limitations. Even so, this design may favor a narrow set of faults or behaviors. It would be useful to know how agent rankings and mutation scores change when mutants have broader kill sets or reflect a wider range of fault types.
  3. The benchmark is based on PROGRAMBENCH and currently covers command-line applications written mainly in Rust, Go, and C/C++. It does not evaluate applications in interpreted languages or other settings such as web services. This limits how far the results can be generalized, although the authors note that the pipeline could be extended to languages such as Python and TypeScript.
  4. The paper shows examples where tests miss combinations of behaviors, but it does not provide a systematic analysis of why this happens. For example, it is unclear whether agents miss these cases because of limited test-generation strategies, the size of the specifications, or the prompts and tools they use. More analysis could help identify ways to improve test generation.
Questions for authors
  1. What changes would be needed to extend the TestSuiteBench construction pipeline to GUI applications or web APIs, where behavior may depend on sequences of actions or state changes?
  2. The paper aims to create mutants that are killed by a small number of reference tests, and reports that 73% are killed by exactly one. The authors also audit mutants to avoid ones that depend on specific hard-coded inputs. Could you test how the results change with mutants that have broader kill sets or represent a wider range of real-world faults? This would help clarify how much the benchmark’s scores depend on the singleton-kill-set design.
  3. The examples show that generated tests can cover individual requirements but miss combinations of behaviors. Have you tested whether different prompts, test-generation strategies, or agent tools help address this problem?
1 comment

We thank the reviewer for the thorough assessment of our work. Below, we address each of the points the reviewer raised.

W1: Is it realistic to use the detailed specifications as input?

Yes, please refer to the global answer (G2), where we highlight (i) the practical implications of a large-spec-based development, and (ii) that we carefully design it to make it more natural than prior benchmarks.

W2 & Q2: Is the mutation score representative of a wider range of real-world faults?

Yes, please refer to the global response (G1), where we provided a new experimental result showing a strong correlation between our mutation score and real-bug detectability.

W3 & Q1: Elaborate on how it could be extended to other domains.

The high-level idea of our benchmark construction is to prepare a consistent set of specifications, mutants, and a reference implementation and test suite. In particular, specifications, mutants, and reference implementations should capture the same input-output behavior. This pattern can be applied beyond CLI programs to domains such as GUI applications and web APIs: Instead of defining inputs in terms of command-line flags and arguments, inputs can be (a sequence of) click or drag operations in GUI applications and arguments to API endpoints for web APIs. For outputs, instead of stdout and stderr outputs, the tests and specification can state assumptions about visible elements in GUI applications and returned values or observable state changes in web APIs. We include a discussion of such extensions in our discussion and limitations section (See Section 5).

W4: Provide more systematic analysis of failure cases.

Yes, please refer to the global answer (G3). We analyzed the surviving mutants and found that the generated test suites often exercise relevant code, but fail to trigger mutated behavior, likely due to insufficient input diversity. When differences do surface, tests often detect them.

Q3 & W4: Explore different test generation approaches.

We provide the following new results. If there are some other specific experiments that the reviewer is interested in, we are happy to evaluate them.

Explicitly instructing to create executable/mutants. As we found in Table 3, Opus (the best-performing model) often creates executables and mutants to check the test. We additionally investigated whether an explicit instruction to create them could improve the mutation score for other models. As shown in the new entry of Table 3 (also pasted in this reply), we found that this strategy slightly decreased the mutation score (37% -> 34%) for Sol. We observe that the total time/cost remains the same, which implies the model spent less effort generating the test suite itself; in fact, the number of generated test functions substantially decreased (on average 125 -> 98). Therefore, executable/mutation-based test generation is not a universal solution, and a deeper look into its influence is left as an interesting avenue for future work.

Table: the new result, “Sol + prompt”, explicitly asks the model to create executables and mutants to check its test suite.

Model / settingExecutable creationMutant creationMutation score (%)
Sol4037.7
Sol + prompt13610334.2

Multi-turn evaluation. In practice, specifications may evolve, requiring iterative test updates. We simulate such a use case by revealing the specification in five increments, asking the agent to update its previous test suite each time. As shown in the new Table 7, we found that this improves mutation score ($\Delta=9.2pp),particularlycontributingforlargespecifications,weakeningthenegativecorrelationbetweenspecsizeandmutationscore(9.2 pp), particularly contributing for large specifications, weakening the negative correlation between spec size and mutation score (\rho=-0.42$ to −0.27-0.27). However, cost increases by 3.5×3.5\times, time by 1.4×1.4\times, and validity drops by 6.3%. Importantly, xhigh reasoning effort of the same model achieves a comparable mutation score (46.5%) at a lower cost ($2.3) in a single-turn setup, suggesting that reasoning effort solves the cost-quality tradeoff better than running the model many times with a focused set of requirements.

Table: Five iterations with Sol at medium reasoning effort. Changes are relative to the main setup.

SettingMutation score (%)Validity (%)# testsCost ($)Time (min)
Multi-turn46.988.13133.1311.0
Change+9.2 pp-6.3 pp2.5×3.5×1.4×
Reviewer_4Sx3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. This paper finds that even frontier coding agents struggle to generate strong test suites given the specifications, achieving mutation scores below 60%, despite their suites being largely valid and 88% of their tests passing on the reference implementation. This paper also shows that generated test suites often lack relevant input combinations and are smaller than reference test suites.

Critical strengths
  • Seems the first benchmark for repository-scale test suite generation given natural-language specifications. And tested it across 6 coding agents.

  • During constructing this benchmark, this paper also proposes a pipeline for producing specifications from existing test suites, refines them by attributing agent failures to specification defects, and generates mutants that each violate a single requirement.

Critical weaknesses

W1: No solid ground truth because the specifications were produced by agents, mutated by agents, and extended by agents. Morover, GPT-5.6 Sol, the agent for constructing the dataset, is one of the evaluated agents.

W2: Not sure the difficulty distribution for the mutants generated by agents because the mutation kill rate may not accurately measure how strong the generated tests against real bugs.

Questions for authors
  1. Please refer to W1 above.

  2. Please refer to W2 above.

2 comments

We thank the reviewer for the thorough assessment of our work. We want to highlight that while both the specifications and mutants were generated by agents, they were extensively validated in our ablation studies. In particular, we ensure that for all mutants, there exist differentiating tests and that they only assert properties described in the specification.

W1 & Q1: Could there be an advantage for Sol because it was involved in the artifact creation process?

We did not observe any signs of Sol having a notable advantage in either its absolute or cost-normalized performance (see Figure 5). Further, our process involves significant manual curation and programmatic quality control, which we believe should limit the effect of the generating model. Please refer to the global response (G1, G2) for more discussion on the artifact quality. As the strongest signal, we observe that our mutation score has a strong correlation with real-bug detectability, as also discussed in the next point.

W2 & Q2: Does the mutation score correlate with the ability to catch real reported bugs?

Yes, please refer to the global response (G2), where we provided a new experimental result showing a strong correlation between our mutation score and real-bug detectability.

Thank you for updating your review. To address your updated points in more detail:

W1: The ground truth is each application’s original, i.e., human-written implementation. Validity is measured against it, and every mutant must be killed by at least one reference test and change only behaviour the specification requires.

To test for construction bias, we use the 1,686 mutants derived from real GitHub fixes (see global response G1). On this subset, we observe a strong correlation between this and our reported mutation score. (Pearson 0.97, Spearman 0.94),

Follow-up on W1. Prompted by the reviewer’s (now deleted) comment, we ran spec and mutant generation using Opus 5.5 on 15 instances randomly sampled from those that did not require manual cleanup (Section 2.4). We then ran the test generation three times with Sol for both the original, Sol-generated, and new, Opus-generated instances.

We observed largely identical mutation scores across the two sets, with only a mean difference of 1-3% and overlapping score ranges across the three runs. Although limited to a small subset of instances, the results provide further evidence that there is no significant bias toward the model used for benchmark construction.

Table: The score range for original (Sol-built) and new (Opus-built) sets of three evaluation.

Evaluated modelOriginal benchmarkOpus-built benchmark
Sol39.2%–41.3%38.1%–39.6%
Opus56.2%–59.4%59.0%–62.5%

W2. Difficulty-distribution Beyond the rank correlation (Pearson 0.97, Spearman 0.94), absolute kill rates are slightly lower on the more specific synthetic mutants: Δ=6\Delta = 6% on average across six models. Please refer to Figure 8 in the revised PDF.

We hope these discussions address your concerns. We would be happy to discuss any remaining questions or concerns.

Reviewer_aedzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how models (and coding agents) perform on full test suite generation for complex software, comparing their ability to generate tests and implementations from specifications. The authors propose a new benchmark, TestSuiteBench, in which models can be evaluated in this setting. It consists of 165 open-source applications, each with a specification, a reference implementation, a reference test suite, and mutants. The authors demonstrate that even frontier models struggle to generate strong test suites. The analysis suggests that specification ambiguity alone does not explain the low performance, as models generate mostly valid tests and implementations. Moreover, providing the reference executable improves test validity but barely changes the mutation score in the evaluated setting. The gap seems related to test design, with generated suites missing relevant behaviors and input combinations.

Critical strengths
  1. The paper is well written and easy to read. Overall, the empirical findings are supported by evidence and explanation. The problem is very relevant, and the paper takes a step forward in assessing automatic test generation: from more "local" testing to full test suites generated from just a specification.
  2. The benchmark construction is described in detail, and the methodology seems carefully designed. The authors motivate why existing developer documentation was insufficient and build a solid pipeline to create detailed specifications. The mutant construction is also explained carefully. Additionally, the authors provide clarity on how they prompted models to generate such data, with detailed prompts in the appendix. Manual checks are also performed to further clean the benchmark.
  3. The experimental framework includes a good variety of models. Generated tests are run three times against the reference executable to identify flaky tests. The authors carry out several analyses to understand performance: (a) the contrast between validity score and mutation score is a useful insight; (b) providing the executable and observing how these two metrics change is also informative.
Critical weaknesses
  1. The authors acknowledge that the specifications are more detailed than typical developer documentation, as they are generated from tests. However, their naturalness and similarity to independently written requirements are not directly assessed. Some specification excerpts are provided, but these are not enough to understand whether the complete specifications represent realistic scenarios or expose unusually specific details from the reference tests. The manual checks address correctness and consistency, but do not establish naturalness. Appendix B shows that details such as exact source locations appeared during construction and were removed; this makes it relevant to understand whether subtler test-specific details remain.
  2. The error analysis could be improved with more actionable insights. A more detailed breakdown of why mutants are not killed across the benchmark would be interesting. For example, do models omit the relevant behavior, choose inadequate inputs, or fail to assert the right output? Are the main weaknesses boundary conditions or interactions between requirements?
  3. "Frontier agents can build software they cannot verify" is a strong statement. I acknowledge that the reference pass rate is high, but that does not mean the full application is built correctly. Also, a low mutation score shows that models struggle to detect the benchmark's injected faults; it does not directly establish whether they detect errors in their own generated implementations. It would be interesting to evaluate whether valid generated tests detect errors in generated implementations that the reference suite identifies as faulty.
  4. While the mutants are carefully designed, the paper would benefit from a demonstration that they represent meaningful failure scenarios (i.e., plausible bugs). This would increase confidence in the metric and in the overall paper.
  5. Many of the repositories are well-known open-source projects, so models may have seen the original code, tests, or documentation during pretraining. The paper does not discuss the potential impact of contamination on the results.
Questions for authors
  • Weakness 2, 3 contains questions whose would strengthen the paper.
1 comment

We thank the reviewer for the positive assessment of our work. Below, we address each of the points the reviewer raised.

W1: Do the specs represent realistic scenarios with no unusually specific details?

Yes, please refer to the global answer (G1), where we highlight (i) the practical implications of large-spec-based development, and (ii) that we carefully design it to make it more natural than prior benchmarks.

Importantly, as the reviewer recognized, we manually investigated and removed test-specific details. We provide a longer spec excerpt from the go-mod instance in Figure 17. Refer to it to get better intuition on how rigorously it documents observable behavior, while not having test-specific descriptions. We believe our specifications to be significantly more natural than prior work (Nl2repo-bench).

W2: Provide a breakdown for understanding why mutants survive.

Yes, please refer to the global answer (G3). We analyzed the surviving mutants and found that the generated test suites often exercise relevant code, but fail to trigger mutated behavior, likely due to insufficient input diversity. When differences do surface, tests often detect them.

W3: Please elaborate on the sentence, “frontier agents can build software they cannot verify”.

We intended to highlight the gap between the agents’ high capability at building repository-level code, as seen in the high reference pass rate, while capabilities at building strong test suites lag behind, as seen in our low mutation score. We clarified this statement with a more explicit phrasing as follows: “frontier agents cannot verify software as effectively as they can build it”

W4: Are the mutants representative of plausible bugs?

Yes, please refer to the global response (G2), where we provided a new experimental result showing a strong correlation between our mutation score and the real-bug detectability.

W5: Please discuss potential impact of contamination.

If models had memorized test cases for a known repository, they should be able to reproduce the same test suite without our detailed specification document. As the mutation score is already low with our detailed specs and drops much further when using only the official specs (see Table 5), we believe this is unlikely. We further explore this in a new experiment where we ask the model to generate a test suite using only the repository name and the general instruction, without providing the specification; see the new Table 8 (also pasted below). We again observe a significant drop in mutation score from 55.5% with specification to at most 10.8% without. This indicates that models can not generate strong test suites from memory, making contamination unlikely.

Interestingly, however, we observe a much smaller drop in validity rate in the same setup, still achieving up to 85.1%. Combined with the much smaller count of generated tests, this indicates that the models have a high-level understanding of the key behaviour of these CLIs based on which they design a few reference-passing tests, but not their implementation.

Table: Test generation with only the repository name. Parentheses show differences from the original setup

ModelMutation score (%)Validity (%)# tests
Opus10.8 (-44.7)85.1 (-11.0)170 (-595)
Sol2.2 (-35.1)66.4 (-23.9)26 (-100)

Comments on the paper

4 comments

We appreciate the reviewers for their extensive feedback. We are particularly happy to hear positive assessments of the importance/relevance of our task setup (Aedz, 4Sx3, J7tz), careful construction (Aedz, J7tz), and extensive evaluation (Aedz, J7tz).

We summarize the major updates to the PDF as follows. Changes are marked in blue.

Main sections

  • Table 3: we evaluate Sol by explicitly instructing it to generate executable/mutants to check their test suite.
  • Table 4: we conducted an error breakdown of mutants that survived model-generated test suites.
  • Figure 8: we compared our mutation score with real-bug detectability.
  • Table 7: we evaluate a multi-turn setup, where the specification is revealed to the model in five increments.
  • Table 8: we added a repository-name-only ablation to probe reliance on the supplied specification and discuss contamination.

Appendix

  • Figure 14: we added qualitative analysis to compare our preliminary, test-specific mutants and our final mutants improved via instructing logic-level mutation.
  • Figure 15: we added qualitative analysis to compare mutants before/after iterative refinement.
  • Figure 17: we added a long excerpt of the specification.

Below, we provide answers to questions raised by multiple reviewers.

G1: Are the mutation scores representative of real-world bug detectability?

Strong correlation with real-bug detectability. We collected GitHub issues from the original repositories, filtered for implementation defects violating the specification (excluding dependencies, packaging issues, etc.), and converted the fixing diffs into 1,686 mutants.

Across six models, mutation scores strongly correlate with real-world bug detectability (Pearson=0.97, Spearman=0.94; see Figure 8 in the revised manuscript), supporting the representativeness of our evaluation. This also motivates synthetic mutants: Real bugs yield only 10 mutants per instance on average, with half of instances having fewer than 5. Our synthetic approach covers a wider range of specification violations, generating 74 mutants per instance on average.

We further analyze mutants in Appendix A.6:

  • (Figure 14) Comparing overly test-specific preliminary mutants under the naive “killed by a single test” requirement, with final mutants using logic-level mutation instructions, showing improved quality.
  • (Figure 15) Comparing mutants before and after iterative refinement (Section 2.2), showing more focused mutations.

G2: Is the task, in particular the thorough specification, realistic and adequate?

Yes. We find that our specification is comparable to both realistic use cases and prior work.

Real software does have a large specification. As agents become more capable of writing code, developers increasingly define software behavior in natural language, leaving implementation largely to agents. For example, the emerging trend of specification-driven development (SDD) [1], supported by Gemini Conductor [2], GitHub SpecKit [3], and others [4-6], treats “code as the implementation detail of the specification” [1]. Specifications precede code and extensively document required behavior (e.g., examples from [3] at [7-8]). Our setup closely reflects these use cases.

Our specification is similar in size to prior benchmarks, yet more realistic. Evaluating whether agents can generate test suites from natural language requirements necessitates detailed specifications, as the goal is to measure whether the test can correctly flag behaviour contradicting these specifications. For example, NL2Repo-Bench [9], a repository-level code (not test) generation benchmark, specifies repository structure, classes/functions, inputs, outputs, and code snippets. Its median specification has 5.8k words, versus 5.6k in ours. However, ours specifies observable CLI behavior without prescribing how it should be realized, leaving agents to autonomously design the test suite or repository. We argue that such behavioral requirements are more practical than specifying software design upfront.

G3: Provide an error breakdown of mutants that survived model-generated test suites.

Certainly. We analyze surviving mutants using two indicators: (i) whether generated tests execute the mutated lines (measured by code coverage), and (ii) whether execution produces observable differences (stdout/stderr/return code) between the reference and mutant.

As shown in the new Table 4 (also pasted below), most surviving mutants are reached by both Sol and Opus (≥93.0\geq 93.0%), yet few produce observable differences (≤16.1\leq 16.1%).

This indicates the following three things: (i) the model-generated test suites at least attempt to test relevant behavior, (ii) the majority of the survival occurs because they fail to provide the right inputs to trigger the mutated behavior (e.g., missing the relevant edge cases or flag combinations), and (iii) there are also some cases where they fail in assertion, although the behavioral difference could have been observed.

MetricOpusSol
Reached98.1%93.0%
Observed (among Reached)16.1%7.5%

Thank you once again for your review! As the discussion period is approaching its end, we would greatly appreciate it if you could let us know if there are any remaining concerns or follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.