- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a technique based on MAP-Elites for unit test generation aiming for improve vibe coding. The basic idea is to model unit test generation as a quality–diversity evolutionary search for size-optimized test generation, in which size-annealing UCB invoking LLMs is adopted. This paper also reports an empirical evaluation of the proposed technique.
Critical strengths
- This paper targets an important problem for vibe coding.
Critical weaknesses
- This paper totally ignores the cost of mutation testing. In fact, mutation testing (i.e., to obtain the mutation scores) is one of the most expensive approaches in software testing.
- In the evaluation, this paper treats search based software testing in an inappropriate way. During search based software testing, the major time cost is spent on calculating coverage. However, the time cost of calculating mutation scores in other compared techniques is ignored. In fact, search based software testing can be also based on mutation scores. If we can also ignores the time cost of calculating mutation scores in such a technique using search based software testing, the cost of search based software testing can be almost free.
- Besides search based software testing, concolic execution or dynamic symbolic execution can also be used for generating unit tests. Typically, concolic execution or dynamic symbolic execution is more suitable for processing complex software modules. I think concolic execution or dynamic symbolic execution should have been used as a baseline.
- There is lack of sufficient details of how search based software testing is used in the evaluation. Typically, search based software testing does not aim at obtaining a test suite as smaller as possible but a test suite maximizing coverage or mutation scores. However, when the information of test coverage or mutation scores is available, it is convenient to minimize the test suite via test suite reduction techniques.
- Since the aim of the proposed technique is to improve code generation, I think direct results on how the proposed technique improves code generation should be a better way to demonstrate the effectiveness of the proposed technique. Due to the lack of suitable oracle information in automatically generated tests, coverage or mutation scores alone can hardly serve as a metric for measuring a test suite's capability of fault detection.
- I feel the presentation of the proposed technique in Section 4 is quite difficult to follow, since the presentation seems to be tightly coupled with MAP-Elites.
Questions for authors
Q1: What is the time cost for obtaining the mutation scores? How many mutants are used? Q2: Have you evaluated the real benefits when applying the proposed technique for agentic code generation? Q3: Have you compared with unit test generation based on concolic execution or dynamic symbolic execution?
6 comments
Author response (part 2)
W5.2. Oracle for regression testing.
For the reviewer's concern that automatically generated tests lack an oracle, we want to respectfully clarify that in regression testing, the existing source code itself serves as the oracle. Regression testing is a well-defined problem in software engineering: it is the testing performed to provide confidence that changes do not harm the existing behavior of the software (Yoo and Harman [7]). The behavior of the current version is recorded as the expected output, and later versions are checked against it (Barr et al. [8]). TeSearch follows this definition. Coverage and mutation score are the default metrics for regression unit tests (Fraser and Zeller [9], Lukasczyk et al. [10], Papadakis et al. [6]). TestGenEval [1], itself an ICLR 2025 benchmark paper, generates regression tests and scores them by coverage and mutation score, the same setup as ours.
W4.1. How search-based software testing is used.
We run Pynguin with its defaults (DynaMOSA, branch coverage, mutation-based assertions, its own minimization) and a time limit of 6000 s per module.
The SBST method fails mostly because its adaptation to real projects is weak, and on many modules it cannot run at all because it does not support the project's configuration. No Pynguin release supports Python 3.6 or 3.7, which 42 of the 129 modules need, and on the other 87 it yields a scored suite on 40, stopped on the other 47 by project set-up that it leaves to the user (16 modules, 12 of them unconfigured Django settings), crashes in type resolution, instrumentation or assertion generation (17), the time cap (11) and suites that do not run (3). To enable a better comparison, we manually wrote the scaffolding Pynguin needs on these 12 modules, so that it can run and be compared. Pynguin then produces a passing suite on 6 of them, with a mean hv of 0.065 on the 5 that have a mutation score, whereas TeSearch's model writes this scaffolding into the suite itself. We add the Pynguin configuration and the failure causes to App. O.
W4.2. Test-suite reduction after generation.
We appreciate the reviewer raising this point, and we are happy to provide additional experiments.
Setup. We apply the same Additional Greedy reduction (Yoo and Harman [7]) to every generator's suites. It keeps every covered line and every search mutant the suite kills, and final hv is scored on held-out mutants.
Table R1. 20 randomly sampled modules, GPT-5.4.
| Generator | tests, before reduced | hv, before reduced |
|---|---|---|
| TeSearch | ||
| OpenEvolve | ||
| Beam search | ||
| TeSearch w/o |
The results demonstrate that post-processing reduction does not change the ranking of the methods. Before and after reduction, TeSearch's suites are the smallest and the highest in hv. Reduction removes at most 4.7 tests and 0.008 hv. Generate-then-reduce therefore does not reach TeSearch: after reduction, the closest competitor, OpenEvolve, keeps 28.7 tests at 0.828 hv, against TeSearch's 26.4 tests at 0.868. We summarized these new experiments into a new §5.5.
W6. Presentation of Section 4.
We have added a design rationale to each component of §4, and revised §3, the §5.6 setup and the captions of the main tables and figures, for better readability.
References:
- [1] TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark. Jain, Synnaeve and Rozière. ICLR 2025.
- [2] EvalPlus: Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. Liu, Xia, Wang and Zhang. NeurIPS 2023.
- [3] MuTAP: Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing. Dakhel, Nikanjam, Majdinasab, Khomh and Desmarais. IST 2024.
- [4] ACH: Mutation-Guided LLM-based Test Generation at Meta. Foster, Gulati, Harman, Harper, Mao, Ritchey, Robert and Sengupta. FSE Companion 2025.
- [5] Just et al.: Are Mutants a Valid Substitute for Real Faults in Software Testing? Just, Jalali, Inozemtseva, Ernst, Holmes and Fraser. FSE 2014.
- [6] Papadakis et al.: Mutation Testing Advances: An Analysis and Survey. Papadakis, Kintis, Zhang, Jia, Le Traon and Harman. Advances in Computers 2019.
- [7] Yoo and Harman: Regression Testing Minimization, Selection and Prioritization: A Survey. Yoo and Harman. STVR 2012.
- [8] Barr et al.: The Oracle Problem in Software Testing: A Survey. Barr, Harman, McMinn, Shahbaz and Yoo. IEEE TSE 2015.
- [9] Fraser and Zeller: Mutation-Driven Generation of Unit Tests and Oracles. Fraser and Zeller. IEEE TSE 2012.
- [10] Lukasczyk et al.: An Empirical Study of Automated Unit Test Generation for Python. Lukasczyk, Kroiß and Fraser. EMSE 2023.
I think the authors' response does clarify some issues, but I don't think I will change my opinion on this paper. First, my core comment on whether the proposed technique can boost code generation is ignored. Second, the proposed technique is quite expensive (for both the use of an LLM and the mutation testing), and it is still unclear whether this expensive technique is more beneficial than cheaper ones, especially for improving code generation. Third, I don't think the authors have calculated the cost of mutation correctly. Even with parallel 8 workers, dealing with 60 mutants is still costly. Note that, each mutant is a variant of the module under test, and calculating mutation scores with the 60 mutants means to execute each of 60 variants of the module against each test. So, I don' think the total cost of dealing with 60 mutants can be lower than executing the module once.
Author response (part 1)
We thank the reviewer for the detailed comments. All changes are marked in blue in the revised PDF.
W1, W2 and Q1. Cost of mutation testing.
We appreciate the reviewer for raising this concern, and we would like to respectfully explain that mutation testing is the established way to score and guide test generation, and TeSearch follows this line of work. TestGenEval [1], the benchmark we evaluate on, reports mutation score. EvalPlus [2] keeps the mutants a suite kills as one requirement when it reduces its test suites. MuTAP [3] guides LLM test generation with the mutants that survive, and Meta's ACH [4] does the same in its LLM test generation. Mutant detection is correlated with real-fault detection beyond what coverage explains (Just et al. [5]), which makes mutation score the standard adequacy measure in test generation (Papadakis et al. [6]).
On the number of mutants (Q1), the search does not run the full mutant set. It scores each candidate on a sample of only 60 mutants per module, and these 60 mutant runs execute on 8 parallel workers. The paper states the split into a search fold and a held-out eval fold in §5.1, and the size of 60 in App. F ("Mutants"). The 60 are sampled from a cosmic-ray pool with a median of 233 mutants so that every function and every operator family is represented.
Sampling 60 mutants does not lower the quality of the delivered suites. Every score in Table 1 is measured on the held-out fold (median 173 mutants), which shares no mutant with the 60. On this fold TeSearch reaches a mutation score of 0.920 and an hv of 0.916 with GPT-5.4, the highest of all methods in Table 1.
On the time cost (Q1), we timed TeSearch (Table R6). With the 60 mutants on 8 parallel workers, mutation scoring takes a median of 295 s per module over the 50 calls, 6.2% of a run of 4785 s, against 77.3% for LLM calls. This cost is amortized in the search phase, and using the generated suite afterwards incurs only the cost of test execution, which is negligible. Every search method in Table 1 scores candidates with the same harness on the same mutants, so the comparison is cost-matched.
Table R6. TeSearch, GPT-5.4, 20 randomly sampled modules.
W3 and Q3. Concolic and dynamic symbolic execution.
We add a DSE baseline, CrossHair, and it falls far short of TeSearch. Mature DSE tools target statically typed languages, including C, Java and .NET (KLEE, SPF, Pex). DSE is rarely used for Python because dynamic typing and C-implemented built-ins make its semantics hard to encode for a constraint solver. CrossHair is the best-maintained DSE tool for Python that we could find.
Table R7. Performance of CrossHair. 6000 s time limit.
The traditional tools deliver a scored suite on 40 to 46 of the 129 modules, whereas TeSearch delivers one on all 129, because an LLM-based generator adapts to each project where a tool needs a supported interpreter and hand-written set-up. Where CrossHair does deliver a suite, it falls far short with only 0.069 hv. Of the 1735 functions CrossHair tries, 689 yield a test. Its sandbox rejects 15 modules whose import writes to disk or starts a process (403 functions), and 32 more functions fail at import or when their annotations are evaluated. 180 cannot be called on their own, and it cannot build valid arguments for 116 (file handles, pytest nodes, sympy objects with invariants). 208 use their full time without an example, 90 end earlier without one, and 17 fail for other reasons. TeSearch meets the same obstacles by writing code: its suites configure the framework and build stand-ins for such objects.
W5.1 and Q2. Benefit for code generation.
We want to respectfully clarify the scope of our paper. TeSearch generates regression unit tests, and our downstream is not vibe coding. The downstream of regression testing is every task where existing behavior must be preserved, such as library or legacy-code migration and the updates a self-evolving agent makes to its own harness. We have evaluated TeSearch on both and shown it effective in §5.6. It improves the migration pass rate from 0.69 with no suite to 0.92 (§5.6.1), and the share of behavior-changing agent revisions rejected from 0.253 with the repository's own tests to 0.599 (§5.6.2). Both tasks are scored by task-specific signals such as pass rate and detection rate.