acceptodds
Under review as a conference paper at ICLR 2027

Small but Mighty: Quality–Diversity Evolutionary Search for Size-Optimized Unit Test Generation

Abstract

Reliable coding agents require verification signals that are not only strong but cheap enough to execute repeatedly within agentic loops. Yet existing unit-test generation methods optimize coverage or mutation score while leaving suite size uncontrolled, often producing increasingly redundant tests. We formulate size-optimized unit test generation as a quality-diversity problem: maximize behavioral coverage while discovering compact suites along the quality–size frontier. We introduce TeSearch, an inference-time evolutionary search that partitions test suites into size niches via MAP-Elites and maintains a Pareto frontier over coverage and mutation score within each niche. A size-annealing UCB allocates generation budget across niches, prioritizing quality early and progressively favoring smaller suites once quality is secured. Crucially, quantitative control through an LLM is itself stochastic: requesting k tests does not reliably produce k tests. TeSearch therefore models the LLM as a noisy actuator through an online landing kernel and compensates for its size-instruction bias before generation. Across 129 TestGenEval modules and three frontier LLMs, TeSearch outperforms conventional inference-time scaling and coding agents, achieving higher hypervolume with substantially smaller suites. The optimized suites further improve test-guided cross-version migration and provide fast, effective acceptance checks for self-evolving agents.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
test generation, quality-diversity optimization, inference-time search, MAP-Elites, LLM agents, multi-armed bandits, code generation, multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5kw9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a technique based on MAP-Elites for unit test generation aiming for improve vibe coding. The basic idea is to model unit test generation as a quality–diversity evolutionary search for size-optimized test generation, in which size-annealing UCB invoking LLMs is adopted. This paper also reports an empirical evaluation of the proposed technique.

Critical strengths
  • This paper targets an important problem for vibe coding.
Critical weaknesses
  • This paper totally ignores the cost of mutation testing. In fact, mutation testing (i.e., to obtain the mutation scores) is one of the most expensive approaches in software testing.
  • In the evaluation, this paper treats search based software testing in an inappropriate way. During search based software testing, the major time cost is spent on calculating coverage. However, the time cost of calculating mutation scores in other compared techniques is ignored. In fact, search based software testing can be also based on mutation scores. If we can also ignores the time cost of calculating mutation scores in such a technique using search based software testing, the cost of search based software testing can be almost free.
  • Besides search based software testing, concolic execution or dynamic symbolic execution can also be used for generating unit tests. Typically, concolic execution or dynamic symbolic execution is more suitable for processing complex software modules. I think concolic execution or dynamic symbolic execution should have been used as a baseline.
  • There is lack of sufficient details of how search based software testing is used in the evaluation. Typically, search based software testing does not aim at obtaining a test suite as smaller as possible but a test suite maximizing coverage or mutation scores. However, when the information of test coverage or mutation scores is available, it is convenient to minimize the test suite via test suite reduction techniques.
  • Since the aim of the proposed technique is to improve code generation, I think direct results on how the proposed technique improves code generation should be a better way to demonstrate the effectiveness of the proposed technique. Due to the lack of suitable oracle information in automatically generated tests, coverage or mutation scores alone can hardly serve as a metric for measuring a test suite's capability of fault detection.
  • I feel the presentation of the proposed technique in Section 4 is quite difficult to follow, since the presentation seems to be tightly coupled with MAP-Elites.
Questions for authors

Q1: What is the time cost for obtaining the mutation scores? How many mutants are used? Q2: Have you evaluated the real benefits when applying the proposed technique for agentic code generation? Q3: Have you compared with unit test generation based on concolic execution or dynamic symbolic execution?

6 comments
AuthorsOpenReview ↗

Author response (part 1)

We thank the reviewer for the detailed comments. All changes are marked in blue in the revised PDF.

W1, W2 and Q1. Cost of mutation testing.

We appreciate the reviewer for raising this concern, and we would like to respectfully explain that mutation testing is the established way to score and guide test generation, and TeSearch follows this line of work. TestGenEval [1], the benchmark we evaluate on, reports mutation score. EvalPlus [2] keeps the mutants a suite kills as one requirement when it reduces its test suites. MuTAP [3] guides LLM test generation with the mutants that survive, and Meta's ACH [4] does the same in its LLM test generation. Mutant detection is correlated with real-fault detection beyond what coverage explains (Just et al. [5]), which makes mutation score the standard adequacy measure in test generation (Papadakis et al. [6]).

On the number of mutants (Q1), the search does not run the full mutant set. It scores each candidate on a sample of only 60 mutants per module, and these 60 mutant runs execute on 8 parallel workers. The paper states the split into a search fold and a held-out eval fold in §5.1, and the size of 60 in App. F ("Mutants"). The 60 are sampled from a cosmic-ray pool with a median of 233 mutants so that every function and every operator family is represented.

Sampling 60 mutants does not lower the quality of the delivered suites. Every score in Table 1 is measured on the held-out fold (median 173 mutants), which shares no mutant with the 60. On this fold TeSearch reaches a mutation score of 0.920 and an hv of 0.916 with GPT-5.4, the highest of all methods in Table 1.

On the time cost (Q1), we timed TeSearch (Table R6). With the 60 mutants on 8 parallel workers, mutation scoring takes a median of 295 s per module over the 50 calls, 6.2% of a run of 4785 s, against 77.3% for LLM calls. This cost is amortized in the search phase, and using the generated suite afterwards incurs only the cost of test execution, which is negligible. Every search method in Table 1 scores candidates with the same harness on the same mutants, so the comparison is cost-matched.

Table R6. TeSearch, GPT-5.4, 20 randomly sampled modules.

ComponentMedian per module (s)Share of run
LLM calls370077.3%
Suite execution + coverage3437.2%
Mutation scoring, 50 calls (60 mutants, 8 parallel workers)2956.2%
Mutant generation, once23.10.5%
Full run4785100%

W3 and Q3. Concolic and dynamic symbolic execution.

We add a DSE baseline, CrossHair, and it falls far short of TeSearch. Mature DSE tools target statically typed languages, including C, Java and .NET (KLEE, SPF, Pex). DSE is rarely used for Python because dynamic typing and C-implemented built-ins make its semantics hard to encode for a constraint solver. CrossHair is the best-maintained DSE tool for Python that we could find.

Table R7. Performance of CrossHair. 6000 s time limit.

Generatormodules with a suitesizecovmuthvtime (s)
CrossHair46/12910.50.4010.1290.0693820
Pynguin40/12920.40.4640.1570.0905874
TeSearch, GPT-5.4129/12938.60.9950.9200.9165980

The traditional tools deliver a scored suite on 40 to 46 of the 129 modules, whereas TeSearch delivers one on all 129, because an LLM-based generator adapts to each project where a tool needs a supported interpreter and hand-written set-up. Where CrossHair does deliver a suite, it falls far short with only 0.069 hv. Of the 1735 functions CrossHair tries, 689 yield a test. Its sandbox rejects 15 modules whose import writes to disk or starts a process (403 functions), and 32 more functions fail at import or when their annotations are evaluated. 180 cannot be called on their own, and it cannot build valid arguments for 116 (file handles, pytest nodes, sympy objects with invariants). 208 use their full time without an example, 90 end earlier without one, and 17 fail for other reasons. TeSearch meets the same obstacles by writing code: its suites configure the framework and build stand-ins for such objects.

W5.1 and Q2. Benefit for code generation.

We want to respectfully clarify the scope of our paper. TeSearch generates regression unit tests, and our downstream is not vibe coding. The downstream of regression testing is every task where existing behavior must be preserved, such as library or legacy-code migration and the updates a self-evolving agent makes to its own harness. We have evaluated TeSearch on both and shown it effective in §5.6. It improves the migration pass rate from 0.69 with no suite to 0.92 (§5.6.1), and the share of behavior-changing agent revisions rejected from 0.253 with the repository's own tests to 0.599 (§5.6.2). Both tasks are scored by task-specific signals such as pass rate and detection rate.

AuthorsOpenReview ↗

Author response (part 2)

W5.2. Oracle for regression testing.

For the reviewer's concern that automatically generated tests lack an oracle, we want to respectfully clarify that in regression testing, the existing source code itself serves as the oracle. Regression testing is a well-defined problem in software engineering: it is the testing performed to provide confidence that changes do not harm the existing behavior of the software (Yoo and Harman [7]). The behavior of the current version is recorded as the expected output, and later versions are checked against it (Barr et al. [8]). TeSearch follows this definition. Coverage and mutation score are the default metrics for regression unit tests (Fraser and Zeller [9], Lukasczyk et al. [10], Papadakis et al. [6]). TestGenEval [1], itself an ICLR 2025 benchmark paper, generates regression tests and scores them by coverage and mutation score, the same setup as ours.

W4.1. How search-based software testing is used.

We run Pynguin with its defaults (DynaMOSA, branch coverage, mutation-based assertions, its own minimization) and a time limit of 6000 s per module.

The SBST method fails mostly because its adaptation to real projects is weak, and on many modules it cannot run at all because it does not support the project's configuration. No Pynguin release supports Python 3.6 or 3.7, which 42 of the 129 modules need, and on the other 87 it yields a scored suite on 40, stopped on the other 47 by project set-up that it leaves to the user (16 modules, 12 of them unconfigured Django settings), crashes in type resolution, instrumentation or assertion generation (17), the time cap (11) and suites that do not run (3). To enable a better comparison, we manually wrote the scaffolding Pynguin needs on these 12 modules, so that it can run and be compared. Pynguin then produces a passing suite on 6 of them, with a mean hv of 0.065 on the 5 that have a mutation score, whereas TeSearch's model writes this scaffolding into the suite itself. We add the Pynguin configuration and the failure causes to App. O.

W4.2. Test-suite reduction after generation.

We appreciate the reviewer raising this point, and we are happy to provide additional experiments.

Setup. We apply the same Additional Greedy reduction (Yoo and Harman [7]) to every generator's suites. It keeps every covered line and every search mutant the suite kills, and final hv is scored on held-out mutants.

Table R1. 20 randomly sampled modules, GPT-5.4.

Generatortests, before →\to reducedhv, before →\to reduced
TeSearch28.8→26.428.8 \to 26.40.871→0.8680.871 \to 0.868
OpenEvolve32.7→28.732.7 \to 28.70.836→0.8280.836 \to 0.828
Beam search30.4→26.830.4 \to 26.80.786→0.7830.786 \to 0.783
TeSearch w/o λ\lambda38.6→33.938.6 \to 33.90.823→0.8160.823 \to 0.816

The results demonstrate that post-processing reduction does not change the ranking of the methods. Before and after reduction, TeSearch's suites are the smallest and the highest in hv. Reduction removes at most 4.7 tests and 0.008 hv. Generate-then-reduce therefore does not reach TeSearch: after reduction, the closest competitor, OpenEvolve, keeps 28.7 tests at 0.828 hv, against TeSearch's 26.4 tests at 0.868. We summarized these new experiments into a new §5.5.

W6. Presentation of Section 4.

We have added a design rationale to each component of §4, and revised §3, the §5.6 setup and the captions of the main tables and figures, for better readability.

References:

  • [1] TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark. Jain, Synnaeve and Rozière. ICLR 2025.
  • [2] EvalPlus: Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. Liu, Xia, Wang and Zhang. NeurIPS 2023.
  • [3] MuTAP: Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing. Dakhel, Nikanjam, Majdinasab, Khomh and Desmarais. IST 2024.
  • [4] ACH: Mutation-Guided LLM-based Test Generation at Meta. Foster, Gulati, Harman, Harper, Mao, Ritchey, Robert and Sengupta. FSE Companion 2025.
  • [5] Just et al.: Are Mutants a Valid Substitute for Real Faults in Software Testing? Just, Jalali, Inozemtseva, Ernst, Holmes and Fraser. FSE 2014.
  • [6] Papadakis et al.: Mutation Testing Advances: An Analysis and Survey. Papadakis, Kintis, Zhang, Jia, Le Traon and Harman. Advances in Computers 2019.
  • [7] Yoo and Harman: Regression Testing Minimization, Selection and Prioritization: A Survey. Yoo and Harman. STVR 2012.
  • [8] Barr et al.: The Oracle Problem in Software Testing: A Survey. Barr, Harman, McMinn, Shahbaz and Yoo. IEEE TSE 2015.
  • [9] Fraser and Zeller: Mutation-Driven Generation of Unit Tests and Oracles. Fraser and Zeller. IEEE TSE 2012.
  • [10] Lukasczyk et al.: An Empirical Study of Automated Unit Test Generation for Python. Lukasczyk, Kroiß and Fraser. EMSE 2023.

I think the authors' response does clarify some issues, but I don't think I will change my opinion on this paper. First, my core comment on whether the proposed technique can boost code generation is ignored. Second, the proposed technique is quite expensive (for both the use of an LLM and the mutation testing), and it is still unclear whether this expensive technique is more beneficial than cheaper ones, especially for improving code generation. Third, I don't think the authors have calculated the cost of mutation correctly. Even with parallel 8 workers, dealing with 60 mutants is still costly. Note that, each mutant is a variant of the module under test, and calculating mutation scores with the 60 mutants means to execute each of 60 variants of the module against each test. So, I don' think the total cost of dealing with 60 mutants can be lower than executing the module once.

Reviewer_r5cVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TESEARCH , a search-based method for unit-test generation that iteratively improves test-suite quality and size. It maintains an archive partitioned into size niches, retaining candidates on the Pareto front within each niche and using them to guide subsequent revisions. The writing is often difficult to follow, and I have concerns about the experimental reporting and fairness of the comparisons, as detailed below. Nevertheless, the evaluation is fairly comprehensive and addresses many questions a reviewer would naturally raise. I therefore recommend Weak Accept, while strongly urging the authors to address the writing and experimental concerns.

Critical strengths
  1. The experimental framework is fairly comprehensive and reflects thoughtful engineering.

  2. The paper compares against a broad range of baselines and reports promising results, achieving the highest quality scores among the methods in Table 1.

Critical weaknesses
  1. The main concern is the writing, particularly in the method and experimental sections.

    (1)The core approach is conceptually straightforward: partition candidates by size, maintain a Pareto front within each partition, and use a UCB-inspired rule to select the target size niche for the next generation. However, the method section largely lists what the authors do, with insufficient explanation of why these choices are appropriate and what insights motivate them. A clearer narrative connecting the problem, the design choices, and their rationale would substantially improve readability.

    (2)The two downstream experiments each include a lengthy description of their setup, with some redundancy. Many figure and table captions are also excessively long, making the results harder to navigate.

    (3)More broadly, much of the prose feels formulaic and unnatural, resembling insufficiently edited AI-generated text. This severely undermines the paper’s readability.

  2. I am concerned about the fairness of the computational-cost comparison with coding agents.

    Table 1 reports that TESEARCH outperforms Claude Code. However, matching the number of model calls does not establish that the methods incur comparable token costs. The actual token overhead of TESEARCH’s search procedure relative to the coding agents therefore needs to be reported.

    Figure 7 provides a token-based cost analysis, but excludes both Claude Code and Cursor. I would like the authors to report the actual token consumption of these two coding-agent baselines and include them in the cost comparison.

    In the Claude Opus 4.8 setting, TESEARCH’s gains over Claude Code are 1.6 percentage points in coverage, 6.9 percentage points in mutation score, and 0.085 in hypervolume. In the downstream migration experiment, the improvement after up to five sequential repair attempts is approximately 2 percentage points. If these gains require substantially greater token consumption—for example, more than 10% additional consumption—this would substantially weaken the method’s practical value.

  3. The machinery devoted to controlling test counts may be over-engineered.

    The method devotes considerable machinery to controlling the number of test functions and makes this quantity central to the search structure. I question whether this degree of count control is necessary. In practice, whether a suite contains 32 or 34 test functions is not particularly important in itself; the more meaningful considerations are its testing quality and the amount of test code. The authors should better justify why controlling test-function counts deserves such a central role in the method.

Questions for authors

If the authors can clarify the cost comparison, better justify the role of test counts, I would consider raising my score.

  1. What are the actual token costs of TESEARCH, Claude Code, and Cursor under the same evaluation setting? Do TESEARCH’s gains hold under comparable token budgets?

  2. Why is the number of test functions central to the search? Would a simpler approach that balances testing quality and test-code size achieve similar results?

2 comments
AuthorsOpenReview ↗

Author response (part 1)

We thank the reviewer for the insightful comments. All changes are marked in blue in the revised PDF.

W2 and Q1. Token cost against the coding agents.

We appreciate the reviewer raising this question. We conduct an additional analysis that reports the tokens and time of TeSearch, Cursor and Claude Code.

Setup. We logged tokens and time at every suite a method wrote. We count an agent's requests from its log. The prompt asks the agent to write a complete suite and to run the scoring harness at every request. When an agent stops before 50 requests, we resume the session and instruct it to keep improving the suite.

Table R3. Scaling table on tokens and time. 20 randomly sampled modules. All methods use Opus 4.8. 50 requests per module.

TeSearchCursorClaude Code
Budgethvtestshvtestshvtests
50k tokens0.84726.30.58836.50.39748.3
100k tokens0.87833.00.64336.10.57739.7
500k tokens0.91223.70.77235.70.76638.9
300 s0.58820.40.64936.00.68437.8
600 s0.81526.80.75136.30.75639.4
3,600 s0.91124.40.79936.00.83538.3
Full run0.91521.70.79936.00.83538.5

Table R4. Full-run tokens and time per module.

MethodTokens (k)Wall time (s)
TeSearch5734702
Cursor3,6612578
Claude Code4,5503335

Table R3 shows that TeSearch delivers the highest hv and the smallest suite overall. The agents are ahead in hv only in the first 5 minutes, and TeSearch leads from 10 minutes on. At 50k tokens its hv is 0.847 with 26.3 tests, against 0.588 with 36.5 for Cursor and 0.397 with 48.3 for Claude Code. After the full run it reaches 0.915 hv with 21.7 tests, against 0.799 with 36.0 and 0.835 with 38.5. Table R4 shows that TeSearch also uses the fewest tokens. It uses 573k tokens per module, against 3,661k for Cursor and 4,550k for Claude Code. An agent runs as one continuing session and reads its whole history at every request, so most of its tokens are cache reads. Each TeSearch call is an independent short prompt, so it uses fewer tokens. The migration gain costs no tokens beyond Table R4, because the guiding suite is generated once per module and reused in every repair attempt. We summarized this experiment into a new App. H.2.

W3 and Q2.1. Why the number of test functions is central to the search.

Great question! This is a design choice we made on purpose, and we are happy to explain why.

We agree that whether a suite has 32 or 34 tests matters little in itself. TeSearch does not control the exact count: each niche covers a range of sizes, with an upper bound 1.22 times its lower bound, so suites of 32 and 34 tests share a niche or sit in adjacent ones, and within a niche suites compete on coverage and mutation score.

We centre the search on the number of tests for two reasons. First, in traditional software testing, the number of test cases is the default measure of suite size (Harrold et al. [1], Namin and Andrews [2], Inozemtseva and Holmes [3]). The number of tests also has a strong correlation with lines of code (rank correlation 0.80 to 0.90, Table 11 in App. I). Second, it is hard for an LLM to control how many lines it writes. Studies of length instructions reach the same conclusion: most models miss a requested length (LIFEBench [4], Yuan et al. [5], Ruler [6]), and they follow counts of structural units such as sentences far better than fine-grained lengths (Retkowski and Waibel [7]).

Setup. We replace the number of tests with lines of code (LOC) throughout TeSearch: the size groups are defined by LOC, each call requests a number of lines, and the landing kernel corrects the bias of that request.

Table R8. Landing error after bias correction, and the final suites. 20 randomly sampled modules, GPT-5.4.

Size the search asks forTypical deviation from the requestCalls landing in the aimed size grouphvtestsLOC
Number of tests±10%69.8%0.87128.8316
Lines of code±25%35.6%0.81230.4377
Scalarized (simple approach)––0.75732.5293

When LOC is the target of the search, the deviation from the requested size becomes much larger and the landing accuracy much lower. The typical deviation rises from ±10% to ±25%, and the share of calls that land in the aimed size group falls from 69.8% to 35.6%. As a result, the LOC version ends with lower hv (0.812 against 0.871) and with more test code (377 lines against 316) at about the same number of tests (30.4 against 28.8). We add this motivation to §3 ("Size as the behavior descriptor") and this experiment to App. M (Table 16).

AuthorsOpenReview ↗

Author response (part 2)

Q2.2. Why a simpler approach that balances quality and size does not achieve similar results.

This is a great question, and it goes to the core motivation of our design. Size-optimized test generation is a multi-objective problem, and a single-objective formulation entangles size with quality.

Setup. Scalarized (last row of Table R8) keeps one pool of suites and ranks them by the single score hv−λz\mathrm{hv} - \lambda z, where zz is the suite's LOC normalized to [0,1][0, 1] in log space. At every call, it asks the LLM to revise the highest-scoring suites into one with higher coverage and mutation score and fewer lines of test code. λ\lambda is the same as in TeSearch: λ=max⁡(0, hvbest−H)\lambda = \max(0,\ \mathrm{hv}^{\mathrm{best}} - H).

Dropping our multi-objective design for a single objective lowers the result further (Table R8). Scalarized and the LOC version of TeSearch both measure size in LOC and differ only in this design, and Scalarized reaches 0.757 hv against 0.812 for the LOC version. The reason is that size and quality pull in opposite directions: larger suites tend to have higher quality, and smaller suites lower (Namin and Andrews [2], Inozemtseva and Holmes [3]). A single mixed score therefore cannot improve both. A step that shortens the suite tends to lower hv, and a step that raises hv tends to lengthen the suite, so the search trades one for the other and gets stuck in a local optimum. This conflict is why TeSearch models the problem as a multi-objective evolutionary search: size only decides which niche a suite competes in, and within a niche suites compete on quality alone, so quality is never traded for size.

W1. Writing.

We appreciate the reviewer pointing out that our presentation can be improved, and we have revised the paper to improve readability. In §4, each component now has a design rationale, which explains why the choice is appropriate and names the ablation that tests it. We have shortened both downstream setups and the captions of the main tables and figures. We have also revised the prose of §3 and the §5.6 setup.

References:

  • [1] Harrold et al.: A Methodology for Controlling the Size of a Test Suite. Harrold, Gupta and Soffa. TOSEM 1993.
  • [2] Namin and Andrews: The Influence of Size and Coverage on Test Suite Effectiveness. Namin and Andrews. ISSTA 2009.
  • [3] Inozemtseva and Holmes: Coverage Is Not Strongly Correlated with Test Suite Effectiveness. Inozemtseva and Holmes. ICSE 2014.
  • [4] LIFEBench: Evaluating Length Instruction Following in Large Language Models. Zhang et al. NeurIPS Datasets and Benchmarks Track 2025.
  • [5] Yuan et al.: Following Length Constraints in Instructions. Yuan, Kulikov, Yu, Cho, Sukhbaatar, Weston and Xu. EMNLP 2025.
  • [6] Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models. Li, Zhang, Li, Liu, Bai, Luo, Chen and Yang. Findings of EMNLP 2024.
  • [7] Retkowski and Waibel: Zero-Shot Strategies for Length-Controllable Summarization. Retkowski and Waibel. Findings of NAACL 2025.
Reviewer_JJx1OpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

TESEARCH aims to generate Python unit test suites that are effective without containing too many tests. It partitions the search space by suite size and maintains a Pareto front over coverage and mutation score in each size bin. A quality-gated UCB allocates the search budget, while an online landing kernel compensates for the LLM's tendency to miss requested test counts. The paper compares TESEARCH with search methods and coding agents on 129 modules and three models, and examines the resulting suites in cross-version migration and revision acceptance tasks.

Critical strengths
  1. The problem matters in practice. Tests used repeatedly by coding agents need to detect errors without becoming expensive to run.
  2. In Table 1, TESEARCH achieves higher quality scores and delivers smaller suites across all three models. The comparison includes several inference-time search methods and coding agents.
  3. Search and final evaluation use separate mutant sets, reducing the risk of optimizing directly for the reported mutation score. The appendix also measures test-code length and execution time.
Critical weaknesses
  1. The central design lacks a direct comparison. The paper does not compare TESEARCH against a strong generator followed by suite reduction that preserves coverage and mutant kills. It is therefore unclear how much is gained by controlling size during generation rather than reducing a strong suite afterward. GEPA and OpenEvolve differ from TESEARCH in several respects and do not isolate the value of size-based partitioning.

  2. The contribution of the landing kernel is not isolated. The w/o kernel ablation removes count-bias correction and landing-probability weighting while also changing how pulls are counted. Equations (4)–(5) imply that, aside from integer rounding, the online bias estimate mainly changes the number of tests requested. A reasonable fit to landing frequencies does not establish that probability weighting improves target selection. A more informative comparison would retain count correction but replace probability weighting with a UCB that credits each request with its observed reward, then ablate count correction separately (§4.1–4.2, §5.1).

  3. Claude Code, Cursor, and TESEARCH are each given 50 model calls, but this need not equalize token use, feedback opportunities, or total runtime. Appendix H's token-budget comparison excludes the two coding agents. Table 1 also reports averages across modules without paired per-module results.

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Author response (part 1)

We thank the reviewer for the constructive suggestions. All changes are marked in blue in the revised PDF.

W1. A strong generator followed by reduction.

We thank the reviewer for raising this excellent question, and we are happy to provide additional experiments.

Setup. We apply the same Additional Greedy reduction (Yoo and Harman [1]) to every generator's suites. It keeps every covered line and every search mutant the suite kills, and final hv is scored on held-out mutants.

Table R1. 20 randomly sampled modules, GPT-5.4.

Generatortests, before →\to reducedhv, before →\to reduced
TeSearch28.8→26.428.8 \to 26.40.871→0.8680.871 \to 0.868
OpenEvolve32.7→28.732.7 \to 28.70.836→0.8280.836 \to 0.828
Beam search30.4→26.830.4 \to 26.80.786→0.7830.786 \to 0.783
TeSearch w/o λ\lambda38.6→33.938.6 \to 33.90.823→0.8160.823 \to 0.816

The results demonstrate that post-processing reduction does not change the ranking of the methods. Before and after reduction, TeSearch's suites are the smallest and the highest in hv. Reduction removes at most 4.7 tests and 0.008 hv. Generate-then-reduce therefore does not reach TeSearch: after reduction, the closest competitor, OpenEvolve, keeps 28.7 tests at 0.828 hv, against TeSearch's 26.4 tests at 0.868. We summarized these new experiments into a new §5.5.

W2. Isolating the landing kernel.

We appreciate the reviewer pointing out that the ablation could be more fine-grained, and we conduct two additional ablations, each removing one component of the landing kernel and keeping the rest. They show that both components contribute: count-bias correction makes suites land in the aimed niche, which keeps them small, and landing-probability weighting raises hv at the same size.

Setup. (a) w/o landing-probability weighting keeps count-bias correction and credits each request with its observed reward. (b) w/o count-bias correction requests the target size itself and keeps landing-probability weighting. w/o kernel removes both components.

Table R2. The 20 modules of W1, GPT-5.4, 50 calls.

Varianthvsizecalls landing in the aimed niche
Full0.87128.869.8%
(a) w/o landing-probability weighting0.82727.564.9%
(b) w/o count-bias correction0.83034.943.9%
w/o kernel (removes both)0.81238.428.3%

Without count-bias correction (b), calls landing in the aimed niche fall from 69.8% to 43.9%. The model writes more tests than requested, so the misses fall into larger niches, the small niches receive few suites, and the delivered suite grows from 28.8 to 34.9 tests. Without landing-probability weighting (a), suites land as before (64.9% in the aimed niche, 27.5 tests), yet hv falls from 0.871 to 0.827. Removing both components is worst on every column. We add this experiment in App. L.

W3.1. Coding agents under token and time budgets.

We appreciate the reviewer raising this question. We conduct an additional analysis that reports the tokens and time of TeSearch, Cursor and Claude Code.

Setup. We logged tokens and time at every suite a method wrote. We count an agent's requests from its log. The prompt asks the agent to write a complete suite and to run the scoring harness at every request. When an agent stops before 50 requests, we resume the session and instruct it to keep improving the suite.

Table R3. Scaling table on tokens and time. The 20 modules of W1. Opus 4.8.

TeSearchCursorClaude Code
Budgethvtestshvtestshvtests
50k tokens0.84726.30.58836.50.39748.3
100k tokens0.87833.00.64336.10.57739.7
500k tokens0.91223.70.77235.70.76638.9
300 s0.58820.40.64936.00.68437.8
600 s0.81526.80.75136.30.75639.4
3,600 s0.91124.40.79936.00.83538.3
Full run0.91521.70.79936.00.83538.5

Table R4. Full-run tokens and time per module.

MethodTokens (k)Wall time (s)
TeSearch5734702
Cursor3,6612578
Claude Code4,5503335

Table R3 shows that TeSearch delivers the highest hv and the smallest suite overall. The agents are ahead in hv only in the first 5 minutes, and TeSearch leads from 10 minutes on. At 50k tokens its hv is 0.847 with 26.3 tests, against 0.588 with 36.5 for Cursor and 0.397 with 48.3 for Claude Code. Table R4 shows that TeSearch also uses the fewest tokens. It uses 573k tokens per module, against 3,661k for Cursor and 4,550k for Claude Code. An agent runs as one continuing session and reads its whole history at every request, so most of its tokens are cache reads. Each TeSearch call is an independent short prompt, so it uses fewer tokens. We summarized this experiment into a new App. H.2.

AuthorsOpenReview ↗

Author response (part 2)

W3.2. Paired per-module results.

We add the paired per-module results in a new App. N, with a per-module scatter plot (Fig. 8) and medians with 95% bootstrap confidence intervals (Fig. 9). TeSearch has higher hv on most of the 129 modules against the strongest search baseline, coding agents and every ablation on every backbone (two-sided Wilcoxon signed-rank test, Holm-corrected p<0.001p < 0.001 for each), and the highest median hv on every backbone.

Reference:

  • [1] Yoo and Harman: Regression Testing Minimization, Selection and Prioritization: A Survey. Yoo and Harman. STVR 2012.

Thanks to the authors for their responses. They have addressed my concerns and increased my confidence in the paper, so I will raise my confidence score. However, I will keep my original rating of 3. I find the work valuable, but I think its contribution and novelty do not quite meet the bar for a 4. If a 3.5 option were available, I would probably give it 3.5.

AuthorsOpenReview ↗

Thank you!

We appreciate the reviewer's insightful and constructive review. We also thank the reviewer for reading our responses thoroughly and for raising the confidence score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.