- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a technique for measuring behavioral diversity of tests generated for improving vibe coding. The basic idea is to obtain the following relationship between each pair of tests and in a test suite, where the generated code that passes would also pass . The relationships of all pairs of tests in the test suites are used to calculate the behavioral diversity of the test suite. The metric of behavioral diversity is then used for test-suite pruning and expansion.
Critical strengths
- This paper targets at an important problem.
- There is something interesting in this paper.
Critical weaknesses
- The proposed technique seems to be too heavy for calculating the behavioral diversity. Given each failing test in the test suite for , a revised version of would be constructed via LLM invocation and this revised version of would be executed against each test in the test suite for calculating behavioral diversity. Due to this heaviness, I wonder if the overhead of calculating behavioral diversity will be larger than that of the direct repair of against each failing test.
- The evaluation is not satisfactory. There is no evaluation on whether the proposed technique would benefit vibe coding. The whole evaluation is only around test suites. Furthermore, the above overhead is totally ignored in the evaluation, although this is actually a major concern of the proposed technique. To make things even worse, the evaluation does include suitable baselines, such as coverage based techniques.
Questions for authors
Q1: What are the real benefits of the proposed technique? Can the benefits cover the its overhead? Q2: Have you tried the proposed technique in real world LLM-based code generation? Q3: Have you tried alternative techniques in a setting of head-to-head comparison?
2 comments
I agree that adding explicit evaluation of the computational overhead would improve the paper. However, I think my concerns are still there. So, I would not recommend acceptance, although I will not strongly object acceptance.
We appreciate your careful assessment of our work and your emphasis on its practical applicability and computational efficiency. We address your questions below with additional experimental results and clarifications.
Q1: What are the real benefits of BTD, and can they justify its computational overhead?
Thank you for raising this important concern. We agree that the computational overhead should be explicitly evaluated. We have therefore measured the token consumption and computational costs of BTD profiling.
On the sd task, Each round of repair for sd requires 186 targeted repair trajectories. Using GLM 5.2 (Z.ai from OpenRouter) + Codex harness, the total measurement cost is 132.33 dollars.
To reduce this overhead, we additionally explored UniBTD, a lightweight proxy for BTD based on unidirectional repair transfer. Instead of independently estimating bidirectional transfer probabilities for every test pair, UniBTD selects a failing test, repairs the implementation to satisfy it, and groups other tests that are also satisfied by the repaired candidate. This process is repeated for the remaining tests, approximating behavioral redundancy without constructing the full pairwise transfer matrix.
Under the same 70-test selection budget, we obtained the following results across five independent repair runs: UniBTD reduces profiling cost by 77.6% (measured by repair count reduction), while achieving comparable downstream repair performance.
These results demonstrate that our repair-transfer-based framework enables both the full BTD metric and a lightweight unidirectional approximation, offering a practical trade-off between measurement fidelity and computational overhead.
Beyond computational costs, the primary benefit of BTD is reducing human review effort and the volume of test-as-spec feedback required for agentic code repair. Developers often need to inspect generated tests to understand which behaviors they specify and determine whether they contribute meaningful new constraints, even when a trusted oracle provides expected outputs. By identifying a compact set of behaviorally diverse tests, BTD reduces the number of additional specifications requiring human inspection and the amount of redundant feedback supplied to the coding agent.
We will clarify this practical workflow and report costs in the revised paper.
Q2: Have you evaluated BTD in real-world LLM-based code generation?
Our current evaluation focuses on documentation-driven agentic code generation and repair using real-world software benchmarks, including ProgramBench and vLLM. In these settings, coding agents generate implementations from documentation and use executable tests as additional behavioral specifications.Since our findings are applied to real-world software use cases, we believe our findings provide a useful foundation for applying BTD to broader agentic coding workflows.
Q3: Comparison with alternative techniques.
Thank you for this valuable suggestion. We added head-to-head comparisons against random, coverage-guided, and mutation-guided test selection on the sd task, using the same 72-test budget and downstream repair protocol.
Post-hoc analysis shows that these gains include concrete requirements such as non-UTF-8 input handling and duplicate-option validation. Some requirements were indistinguishable under the measured coverage and mutation features but were not interchangeable under observed repairs. These cases illustrate the behavioral significance of the aggregate improvement. We will include these head-to-head comparisons and their matched-budget protocol in the revised evaluation.