- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposesa method, namely CovQValue, for iterative LLM-based test generation inspired by Bayesian exploration. It uses coverage information and per-function reachability estimates to guide test-plan generation and selection. The paper also introduces RepoExploreBench and evaluate CovQValue on two benchmarks using three LLMs. The results show substantial improvements in branch coverage over greedy baselines.
Critical strengths
- The paper addresses a relevant problem in LLM-based test generation.
- The proposed method achieves substantial coverage improvements across two benchmarks.
- The evaluation includes three LLMs and useful ablation studies.
Critical weaknesses
-
The contribution of future-value estimation is not supported by the ablation results. A central motivation of the paper is to account for future exploration opportunities. However, Figure 5(c) reports slightly higher mean coverage when the future-value term is removed ((=0)), compared with the default configuration (88.6 versus 85.5). The component ablation shows a similar result, with coverage changing from 86.5 to 85.5 after adding the Bayesian future-value term. These differences do not establish that the term is harmful, but neither experiment demonstrates a clear benefit. Most of the reported improvement comes from targeting unexplored functions. While multi-step planning may still be useful, the specific contribution of future-value estimation remains unsubstantiated.
-
The probabilistic interpretation of the reachability estimates is unclear. The method maintains a Beta posterior to estimate each function's reachability. Equation (3) assumes that a plan increases the future reachability of functions it enables, even if they are not immediately executed. However, Algorithm 1 increases the posterior's failure count whenever a predicted function is not covered. This also applies to functions predicted to be enabled but not executed. Their estimated reachability therefore decreases, despite the future-value calculation assuming an increase. These mechanisms may represent different aspects of reachability, but the paper does not clearly distinguish them. As a result, the probabilistic meaning of the exploration state remains uncertain.
-
The function-level exploration objective does not fully align with the branch-level coverage goal. Algorithm 1 defines unexplored functions as those for which no line has been executed. Equation (3) then estimates exploration value using this set of functions. Once a function has been executed, it is excluded from the explicit exploration objective, even if many of its branches remain uncovered. This is important because reaching deeper branches is a central motivation of the paper. Although the coverage map may still help the agent discover such branches, the scoring mechanism does not explicitly account for them. The gap between function-level targeting and branch-level coverage is therefore not fully addressed.
-
The empirical evidence for the incremental contribution over existing coverage-guided methods is limited. The main Greedy baseline does not receive explicit coverage feedback, while CovGreedy receives the coverage map but selects candidates randomly. CovQValue additionally uses function targeting, multi-step plans, and candidate scoring. The component ablation helps separate these effects and shows that targeting accounts for most of the improvement. However, coverage-guided targeting is already used in existing approaches such as CoverUp. The comparison with CoverUp uses each tool's native budget, making the results difficult to interpret as a controlled comparison. Moreover, the advantage over CoverUp is relatively small for Gemini 3.8 Flash (103.5 versus 98.0 branches). The evidence for an additional benefit over strong existing approaches is therefore less convincing than the comparisons with the basic greedy baselines suggest.
Questions for authors
- What evidence demonstrates the benefit of future-value estimation beyond targeted immediate-gain scoring?
- What probability does the per-function Beta posterior represent?
- How does CovQValue account for uncovered branches within previously executed functions?
- How does CovQValue compare with CoverUp under matched execution and inference budgets?
- How is the fault-detection effectiveness of the generated scripts assessed?
2 comments
As we are close to the end of the discussion period, we wanted to check if our response addressed your concerns. We went over the future-value evidence, the posterior definition, the matched-budget CoverUp comparison, and the mutation analysis. We would appreciate if you can reconsider your score in light of the new results, and we remain available for any additional question.
In the following response, we tried to answer your questions. The reported numbers are mean covered branches per target on RepoExploreBench (93 targets, 24 executions) unless stated. ± is the 95% confidence margin. And counts are slightly lower than in the paper because we now count only true branch arcs.
W1 / Q1: Benefit of future-value estimation
We compared the method with and without the future-value term on RepoExploreBench, and on synthetic programs built to require lookahead (a chain of 4–8 functions that must be called in order across separate test scripts, with most branches behind the last one). There we know what each plan truly unlocks and can substitute it for the LLM's prediction (60 paired runs). On these programs, every compared method keeps a function as a target until all of its lines have run.
With the true downstream values, the term adds 13.6 branches. With the LLM's predicted values the effect is neutral, which matches Figure 5(c). The bottleneck is prediction quality. We show this bottleneck and show how we can potentially reach better results with our method with better prediction capabilities.
W2 / Q2: What the Beta posterior represents
pf is the probability that a function the LLM expects a plan to execute really gets executed. The inconsistency you describe comes from functions predicted as "enabled", which Algorithm 1 penalized when they were not executed right away. We removed that penalty, so only execution predictions update pf. Coverage barely moves (−0.4 branches on Gemma, 3 seeds), so the results in the paper do not depend on it. We update Algorithm 1 and state this definition.
W3 / Q3: Uncovered branches inside executed functions
We added hybrid targeting. In each round, one candidate plan revisits the executed functions with the most uncovered branches, and its prompt lists those branches.
The gain grows once the unexecuted functions run out, and we will adopt hybrid targeting in the revision. A function-level breakdown of where the branch gains come from is in our response to Reviewer jDfu (W3 / Q3).
W4 / Q4: CoverUp under matched budgets
We reran CoverUp with a cap of 48 LLM calls and 24 executions per target, the budget CovQValue uses.
Under the same budget, CovQValue covers more than twice as many branches on both models, while at its native budget CoverUp needs 2.6 times our LLM calls. CoverUp handles uncovered segments one at a time in file order, and on 55 of 93 targets (Gemma) it ends the matched budget with no passing test. CovQValue instead spends each execution on the plan with the highest expected gain.
Q5: Fault detection
We measured fault detection with mutation analysis. For each of 30 RepoExploreBench modules, sampled across its 9 repositories, we created 50 faulty versions with one small change each. We then ran each tool's tests (generated with Gemma) and counted how many faulty versions they detect. Our scripts have no assertions, so a fault counts as detected when a script's output or exception changes.
When a tool produced no tests for a module, it detects no faults there (6 modules for CoverUp, 14 for Pynguin).
CovQValue detects far more faults than Pynguin and CovGreedy, and is on par with or better than CoverUp's assertion-based tests. Output comparison and assertions are different checks, so that last comparison is approximate. The higher coverage does translate into more faults detected.