acceptodds
Desk rejected at ICLR 2027

BEHAVIORAL TEST DIVERSITY: QUANTIFYING TEST SUITE DIVERSITY THROUGH AGENTIC CODE REPAIR

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

As coding agents make it easier to generate code at scale, ensuring that the resulting code aligns with human intent becomes a central challenge. Testing is widely used to check whether code behaves as intended, with tests serving as executable specifications. The increasing effort required to construct tests has motivated growing use of LLM based test generation, which requires effective objectives to guide test suite construction. Existing objectives do not directly quantify the diversity of behavioral constraints specified by a test suite. High performance under these metrics can therefore coexist with superfluous redundancy of some behaviors and omissions of others, leaving gaps in the testing surface. We introduce Behavioral Test Diversity (BTD), an agent conditioned metric that measures this diversity and provides an objective for test suite optimization. BTD quantifies behavioral overlap by observing whether code repairs guided by one test also satisfy others, without needing predefined behavior categories. The resulting score serves as an objective for preserving behavioral diversity during test suite pruning and expansion. Our experiments show that BTD yields stable scores and preserves relative test suite rankings. Across our evaluations on ProgramBench, vLLM, and ComputeEval, BTD guided pruning reduces the number of tests supplied as additional executable specifications by 61 to 90% while retaining sufficient behavioral guidance for full suite recovery. We also show that BTD guided expansion covers previously unchecked behavior that baseline methods miss. Together, these studies support BTD’s validity and practical utility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic coding, software testing, test diversity, test suite evaluation, program repair, code generation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u7nTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a technique for measuring behavioral diversity of tests generated for improving vibe coding. The basic idea is to obtain the following relationship between each pair of tests tit_i and tjt_j in a test suite, where the generated code that passes tit_i would also pass tjt_j. The relationships of all pairs of tests in the test suites are used to calculate the behavioral diversity of the test suite. The metric of behavioral diversity is then used for test-suite pruning and expansion.

Critical strengths
  • This paper targets at an important problem.
  • There is something interesting in this paper.
Critical weaknesses
  • The proposed technique seems to be too heavy for calculating the behavioral diversity. Given each failing test tt in the test suite for C0C_0, a revised version of C0C_0 would be constructed via LLM invocation and this revised version of C0C_0 would be executed against each test in the test suite for calculating behavioral diversity. Due to this heaviness, I wonder if the overhead of calculating behavioral diversity will be larger than that of the direct repair of C0C_0 against each failing test.
  • The evaluation is not satisfactory. There is no evaluation on whether the proposed technique would benefit vibe coding. The whole evaluation is only around test suites. Furthermore, the above overhead is totally ignored in the evaluation, although this is actually a major concern of the proposed technique. To make things even worse, the evaluation does include suitable baselines, such as coverage based techniques.
Questions for authors

Q1: What are the real benefits of the proposed technique? Can the benefits cover the its overhead? Q2: Have you tried the proposed technique in real world LLM-based code generation? Q3: Have you tried alternative techniques in a setting of head-to-head comparison?

2 comments

We appreciate your careful assessment of our work and your emphasis on its practical applicability and computational efficiency. We address your questions below with additional experimental results and clarifications.

Q1: What are the real benefits of BTD, and can they justify its computational overhead?

Thank you for raising this important concern. We agree that the computational overhead should be explicitly evaluated. We have therefore measured the token consumption and computational costs of BTD profiling.

On the sd task, Each round of repair for sd requires 186 targeted repair trajectories. Using GLM 5.2 (Z.ai from OpenRouter) + Codex harness, the total measurement cost is 132.33 dollars.

To reduce this overhead, we additionally explored UniBTD, a lightweight proxy for BTD based on unidirectional repair transfer. Instead of independently estimating bidirectional transfer probabilities for every test pair, UniBTD selects a failing test, repairs the implementation to satisfy it, and groups other tests that are also satisfied by the repaired candidate. This process is repeated for the remaining tests, approximating behavioral redundancy without constructing the full pairwise transfer matrix.

Under the same 70-test selection budget, we obtained the following results across five independent repair runs: UniBTD reduces profiling cost by 77.6% (measured by repair count reduction), while achieving comparable downstream repair performance.

Sample size 5Score
BTD807.40±3.29807.40 \pm 3.29
UniBTD804.00±2.92804.00 \pm 2.92

These results demonstrate that our repair-transfer-based framework enables both the full BTD metric and a lightweight unidirectional approximation, offering a practical trade-off between measurement fidelity and computational overhead.

Beyond computational costs, the primary benefit of BTD is reducing human review effort and the volume of test-as-spec feedback required for agentic code repair. Developers often need to inspect generated tests to understand which behaviors they specify and determine whether they contribute meaningful new constraints, even when a trusted oracle provides expected outputs. By identifying a compact set of behaviorally diverse tests, BTD reduces the number of additional specifications requiring human inspection and the amount of redundant feedback supplied to the coding agent.

We will clarify this practical workflow and report costs in the revised paper.

Q2: Have you evaluated BTD in real-world LLM-based code generation?

Our current evaluation focuses on documentation-driven agentic code generation and repair using real-world software benchmarks, including ProgramBench and vLLM. In these settings, coding agents generate implementations from documentation and use executable tests as additional behavioral specifications.Since our findings are applied to real-world software use cases, we believe our findings provide a useful foundation for applying BTD to broader agentic coding workflows.

Q3: Comparison with alternative techniques.

Thank you for this valuable suggestion. We added head-to-head comparisons against random, coverage-guided, and mutation-guided test selection on the sd task, using the same 72-test budget and downstream repair protocol.

Score ( sample size 5)
Random794.20±6.50794.20 \pm 6.50
Coverage792.80±6.98792.80 \pm 6.98
Mutation795.60±5.55795.60 \pm 5.55
BTD809.20±1.79809.20 \pm 1.79

Post-hoc analysis shows that these gains include concrete requirements such as non-UTF-8 input handling and duplicate-option validation. Some requirements were indistinguishable under the measured coverage and mutation features but were not interchangeable under observed repairs. These cases illustrate the behavioral significance of the aggregate improvement. We will include these head-to-head comparisons and their matched-budget protocol in the revised evaluation.

I agree that adding explicit evaluation of the computational overhead would improve the paper. However, I think my concerns are still there. So, I would not recommend acceptance, although I will not strongly object acceptance.

Reviewer_kmnEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces an agent-conditioned metric, named Behavioral Test Diversity (BTD), to measure the diversity of behavioral constraints represented by a test suite. BTD estimates pairwise test similarity through cross-test transfer during agentic code repair and aggregates these similarities into a diversity score. The authors empirically evaluate the metric's stability and apply it to test-suite pruning and expansion.

Critical strengths
  • The paper targets an interesting problem.
  • The paper proposes an interesting measurement approach.
  • The empirical results are promising.
Critical weaknesses
  • The connection between mutual repair transfer and behavioral redundancy is not sufficiently justified. Equation (3) defines pairwise similarity as the minimum of the two directional repair-transfer probabilities. However, this choice may fail to recognize redundant tests when their behavioral constraints have an implication relationship. For example, suppose one test requires both behaviors A and B, whereas another requires only A. Repairing A+B may always satisfy A, while repairing A may not satisfy A+B. The proposed similarity can then be zero, yielding a BTD score of two, although the second test introduces no additional constraint once the first is retained. Consequently, removing the redundant test reduces BTD, potentially discouraging its removal during pruning. Since identifying and removing redundant tests is a central motivation of the proposed metric, it remains unclear whether mutual repair transfer adequately captures the notion of behavioral redundancy required for test-suite minimization.

  • The reliability of pairwise similarity estimation under small repair budgets remains uncertain. Since the similarity is defined by the minimum of two empirical probabilities, even moderate true bidirectional transfer rates can frequently produce zero observed similarity. However, the stability experiments mainly examine aggregate BTD scores, while they do not establish that the underlying pairwise similarities are accurately estimated or that test-selection decisions are robust to sampling uncertainty.

  • The empirical comparisons are insufficient to establish the advantages of BTD over existing selection criteria. The most direct BTD-versus-coverage comparison is restricted to one vLLM component and a small number of repair replications. Broader comparisons against random selection, mutation-based selection, and alternative diversity-based approaches are expected. Furthermore, several evaluations use BTD retention or BTD gain as their main quality measures. Although full-suite recovery provides useful independent evidence, it does not by itself establish that the selected tests retain comparable fault-detection effectiveness or generalize to unseen faults.

  • The contribution of BTD to test-suite expansion is not clearly isolated. It is unclear whether BTD directly guides test selection or is primarily used to measure the resulting diversity after generation. Consequently, the reported improvements may reflect the benefits of mutation-guided proposals and iterative repair but not the proposed diversity metric itself.

  • The practical cost and scope of test reduction need further evaluation. The reported reductions primarily concern initially failing tests supplied as additional specifications, while initially passing tests remain as regression constraints. For instance, the sdsd experiment retains 696 of the original 810 tests after pruning, corresponding to a substantially smaller reduction in the complete feedback suite. BTD measurement also requires many independent agent repairs and cross-test executions.

Questions for authors
  • How does BTD handle tests with asymmetric implication relationships?
  • How sensitive are the similarity matrices and selected subsets to the repair-sampling budget?
  • What improvement remains when the mutation-guided repair loop is held fixed and BTD-based selection is removed?
  • What are the computational costs of collecting the repair-transfer matrices?
3 comments

We sincerely thank you for the careful review and constructive suggestions, and particularly for recognizing the significance of our research problem and the potential of our approach. We conducted additional controlled experiments and ablations to address these concerns. Below, we present our response so far to several of the points raised.

W1 / Q1: Justify mutual repair transfer and the choice of symmetrization.

Thank you for highlighting the distinction between asymmetric implication and mutual behavioral overlap. We agree that this distinction deserves further evaluation. Our use of min is a conservative choice that requires transfer in both directions before treating tests as substitutable, reducing the risk of overly aggressive pruning and unreliable downstream repair outcomes.

To validate this choice, we added an ablation comparing min, mean, and max symmetrization under a fixed 70-test budget. We further designed stress tests to evaluate robustness under unfavorable test selections. Specifically, for each symmetrization method, we searched for 70-test subsets with low outgoing repair transfer, meaning whose targeted repairs were less likely to resolve other failing tests. Each stress-test subset was required to satisfy the corresponding 95% diversity-retention criterion. This evaluates whether preserving the measured diversity still provides reliable repair guidance under challenging conditions.

Symmetrization (5 samples)Passing tests 810
Min (stress test)798.20±9.42798.20 \pm 9.42
Mean (stress test)783.80±3.03783.80 \pm 3.03
Max (stress test)766.60±9.86766.60 \pm 9.86

Across five independent runs, min outperforms mean by 14.4 tests and max by 31.6 tests on average under the stress-test setting. These results support our choice of mutual transfer as a conservative criterion for preserving reliable corrective guidance. We will clarify this design trade-off and include the symmetrization ablation and stress-test protocol in the revised paper.

W2 / Q2: Reliability of BTD estimation under limited repair samples.

To further evaluate this, we extended our analysis to six repair samples per target and examined how the BTD score varies with the sampling budget RR.

RRSample sizeBTD Score
1678.33±1.3778.33 \pm 1.37
21573.62±1.3973.62 \pm 1.39
32071.57±0.9171.57 \pm 0.91
41570.37±0.6570.37 \pm 0.65
5669.54±0.4869.54 \pm 0.48
6168.89

As RR increases, the estimated BTD gradually decreases due to reduced finite-sample bias, while its variability across sampled subsets also narrows.

To further validate its downstream effectiveness, we conducted 10 independent repair runs using the selected 72 tests. The repaired implementations passed 808.50±2.72808.50 \pm 2.72 out of 810 tests, achieving complete recovery in 7/10 runs.

Together, these results show that, despite finite-sample variability in BTD estimation, the selected subset provides reliable corrective guidance for downstream repair. We will add this sampling-budget analysis and the repeated downstream-repair evaluation to the revised paper.

W3: Establish advantages over alternative test-selection methods.

We appreciate this suggestion and agree, adding direct comparisons against random, branch-coverage-guided, and mutation-guided selection on the full sd task. All methods select 72 failing tests, retain the same 624 regression tests, and use the same downstream repair workflow.

Score ( sample size 5)
Random794.20±6.50794.20 \pm 6.50
Coverage792.80±6.98792.80 \pm 6.98
Mutation795.60±5.55795.60 \pm 5.55
BTD809.20±1.79809.20 \pm 1.79

Post-hoc analysis shows that these gains include concrete requirements such as non-UTF-8 input handling and duplicate-option validation. Some requirements were indistinguishable under the measured coverage and mutation features but were not interchangeable under observed repairs. These cases demonstrate that BTD's higher test recovery reflects meaningful differences in behavioral requirements..

W4 / Q3: Isolate BTD's contribution to test-suite expansion.

Thank you for raising this important point. The original expansion experiment used BTD to evaluate the resulting suites, rather than explicitly computing it during generation.. To isolate the contribution of the BTD metric itself , we added a controlled experiment where BTD is used explicitly for test selection, while keeping the remaining workflow identical.

Specifically, we first let an agent randomly propose inputs and predict their expected outputs based on the documentation. We then compare these predictions with the current implementation's outputs, retaining inputs that reveal potential behavioral misalignment. From this common candidate pool, we compare BTD-based clustering and selection against random selection, using the same budget of KK tests. Only the selected tests are queried against the reference implementation to obtain verified outputs, which are subsequently used to guide code repair.

Each with 3 samplesK=2K=2K=6K=6K=10K=10K=20K=20
BTD642.67±10.97642.67 \pm 10.97661.33±14.15661.33 \pm 14.15664.00±15.52664.00 \pm 15.52691.33±20.03691.33 \pm 20.03
Random642.67±11.68642.67 \pm 11.68643.00±14.00643.00 \pm 14.00657.67±18.50657.67 \pm 18.50680.00±9.17680.00 \pm 9.17

BTD-based selection matches or outperforms random selection across all evaluated verification budgets, independently of the input-generation process and the shared repair workflow. At K=6K=6, BTD recovers 18.33 more tests on average than random selection under the same verification budget. Moreover, BTD with only 6 verified tests achieves higher average recovery than random selection with 10 verified tests (661.33 vs. 657.67), requiring 40% fewer verifications. These results demonstrate that BTD-based clustering helps prioritize more informative behavioral specifications, achieving better repair outcomes with the same verification effort, or comparable outcomes with fewer verifications.

Since verifying expected outputs represents a key component of human test-review effort, these findings also highlight BTD's potential to reduce human review burden while preserving effective corrective guidance. We will incorporate this controlled comparison and clarify BTD's explicit role in test selection in the revised paper.

W5 / Q4: Clarify the scope and computational cost of test reduction.

We agree that these costs should be explicitly quantified.

Computational overhead. Each round of repair for sd requires 186 targeted repair trajectories. Using GLM 5.2 (Z.ai from OpenRouter) + Codex harness, the total measurement cost is 132.33 dollars.

Lower-cost approximation with UniBTD. To address this overhead, we additionally developed UniBTD, a lightweight proxy for BTD based on iterative, unidirectional repair transfer. Starting from the initial implementation (C0C_0), UniBTD selects a failing test and repairs the implementation to satisfy it. All other failing tests also satisfied by the repaired candidate are grouped together as providing potentially overlapping corrective guidance. The process is repeated for the remaining uncovered tests, allowing us to approximate behavioral groups without collecting independent bidirectional repair-transfer estimates for every test pair.

Under the same 70-test selection budget, we obtain the following downstream repair results across five independent runs:

Sample size 5Score
BTD807.40±3.29807.40 \pm 3.29
UniBTD804.00±2.92804.00 \pm 2.92

UniBTD reduces the measured profiling cost by 77.6% (measured by repair count reduction), while achieving comparable downstream repair performance. These results demonstrate that our repair-transfer-based framework enables both the full BTD metric and a lightweight unidirectional approximation, offering a practical trade-off between measurement fidelity and computational overhead.

We will distinguish failing-test reduction from total-suite reduction in the main text and add the profiling-cost breakdown and UniBTD comparison to the revised paper.

Dear Reviewer kmnE,

Thank you again for your thoughtful and constructive feedback. We would like to briefly follow up on our rebuttal, as your suggestions have motivated several additional experiments and helped us strengthen our evaluation.

In particular, we have addressed your main concerns as follows:

  1. Mutual repair transfer and symmetrization. We added symmetrization ablations and stress tests to justify our conservative use of mutual repair transfer. Under challenging test-selection conditions, min outperforms mean and max by 14.4 and 31.6 passing tests on average, respectively, supporting its effectiveness in preserving reliable corrective guidance.

  2. Reliability under limited repair samples. We examined BTD estimation across different sampling budgets and conducted 10 independent downstream repair runs using the selected 72 tests, achieving complete recovery in 7/10 runs.

  3. Comparison with alternative selection methods. We added direct comparisons against random, branch-coverage-guided, and mutation-guided selection under identical test budgets and downstream repair procedures.

  4. BTD's contribution to test-suite expansion. We conducted a controlled experiment comparing BTD-guided selection against random selection, keeping the input-generation and repair workflows identical. BTD achieves better recovery with only 6 verified tests than random selection with 10, requiring 40% fewer verifications.

  5. Computational cost and practical applicability. We quantified profiling overhead and introduced UniBTD, a lightweight unidirectional approximation that reduces profiling repair counts by 77.6% while maintaining comparable downstream repair performance.

We hope these additional results help address your main concerns. We would greatly appreciate hearing whether there are any remaining questions or concerns we could further clarify.

Thank you again for your time and consideration!

Best, The Authors

Reviewer_Yi8WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Behavioral Test Diversity (BTD), a measure of how many distinct code behaviors a test suite constrains. An agent first writes an initial implementation C0 from documentation alone. For each test t_i that C0 fails, the agent repairs C0 R times until t_i passes with no regressions. The rate at which these repairs also pass another failing test gives P_ij, the similarity is S_ij = min(P_ij, P_ji), and BTD is 1 / min_{π∈Δ} π^T S π (Eq. 4, the maximum diversity of Leinster & Meckes). The authors report BTD stability on five ProgramBench tasks and use BTD for test pruning on sd, vLLM BlockPool, and ComputeEval, for test expansion on sd, and for comparing the "alignment efficiency" of three agents. The main claims are that BTD is stable, that BTD-guided pruning cuts the number of failing tests given as extra specification by 61–90% while still recovering all tests, and that BTD-guided expansion finds behaviors the original tests miss.

Critical strengths
  1. The problem matters. LLM-generated tests make suites grow fast, and human review is costly. The observation that test count differs from behavioral diversity is useful. The sd example, where 12 tests repeatedly check that preview does not modify files (App. E.4), makes this point well.
  2. The idea of measuring behavioral overlap through repair transfer is novel and fits how LLM coding agents are used. The measure needs no predefined behavior catalog, so it applies directly to existing suites. This is an advantage over catalog-based methods such as Paul & Holmes (2026).
  3. The measurement protocol is detailed. Algorithm 2 specifies fresh contexts, frozen candidates, blind evaluation (cross-test results are not shown to the agent), two replays, and a certified MILP tolerance. App. E.2 reports that two replays agree exactly on 1,488 vectors. The repeated measurements on bedtools2 (3 batches, R up to 24) are careful and consistent.
  4. The qualitative examples (bed12tobed6 t0006/t0007, vLLM test_prefill_plp, the sd relative-path issue) help the reader see what BTD captures.
  5. The authors report unfavorable numbers honestly, for example the no-repair-loop baseline runs of 2.945, 1.600, and 0, and the Coverage variant adding no tests.
Critical weaknesses

Key reasons for my recommendation: (1) The pruning results, which give the headline 61–90% number, lack basic baselines such as random subsets, use very few runs, and do not count retained regression tests or the cost of computing BTD. (2) There is no external validation that BTD tracks real behavioral differences, and BTD is confounded with the repair agent's style and possible data contamination. The expansion experiment also does not use BTD as its guide and is evaluated only with BTD.

  1. The pruning experiments lack baselines, so the gains cannot be attributed to BTD.

    • On sd (Sec. 5.1.1, App. D.3) there is no baseline. The paper reports one joint repair with the 72 BTD-selected tests reaching 810/810. There is no comparison with random subsets of equal size, greedy coverage or mutation minimization, or stratified sampling by test file.
    • Existing evidence suggests random subsets may work equally well. In App. Fig. 8(b), 416/435 test pairs among the 30 bedtools2 failing tests already reject all 720 single-test repair candidates. A few tests often give strong constraints, so 72 random failing tests may also recover all of sd.
    • On vLLM (Sec. 5.2, Table 1) the only baseline is a "coverage-optimal" set of 7 tests, with coverage measured on the reference implementation. There is no random baseline and no "one random test per connected component" baseline. The latter matters because App. C.3 states that the non-singleton components of S are cliques, so the BTD selection equals "one per component." If a random pick per component also recovers all tests, then the cluster structure does the work and the MILP aggregation is not needed.
    • The budget may be cherry-picked. The text says the budget is increased progressively until "a budget of seven tests," which is the smallest budget at which BTD recovers. Please report recovery rate per budget for both methods.
    • Sample sizes are small: 1 recovery run on sd and 3 per condition on vLLM and ComputeEval, with no confidence intervals or tests.
    • Fix: add random subsets of equal size (at least 10 seeds), coverage/mutation greedy minimization, and one-random-per-component baselines on sd, vLLM, and bedtools2. Report recovery-vs-budget curves with confidence intervals.
  2. The "61–90% reduction" overstates the savings and ignores the cost of BTD.

    • The denominator covers only failing tests. All tests G that C0 passes are kept as regression constraints. On sd, 186 failing tests drop to 72, but the agent still receives 72 + 624 = 696 tests (App. D.3, "Cumulative feedback contains 696 checks"). That is only about a 14% reduction relative to 810. The stated motivation (Sec. 2.1) is human review and execution cost, and the tests in G also need review and execution.
    • The 90% figure cannot be checked from the main text. Table 2 does not give the number of failing tests per ComputeEval task. From "30/30" for CUDA/104 and "three repairs per single-test selection" in App. F.2, I infer that CUDA/104 has 10 failing tests. Please state n and the reduction for each task.
    • Computing BTD needs n × R full repair trajectories (744 on sd, 720 for bedtools2 calibration) plus evaluating each candidate on all of T. These trajectories already produce code that passes each test. The paper reports no tokens, cost, or wall-clock time. BTD also requires trusted expected outputs for all tests before repair can run, so review has already happened before pruning. Please describe a workflow in which BTD saves cost, and report its compute overhead.
  3. The expansion methods are not guided by BTD, and they are evaluated only with BTD.

    • In Sec. 5.1.2 and App. D.4, the "BTD-guided (Reach/Kill / Flat / Coverage)" variants share a repair loop: after each accepted test, the agent repairs, and the next round targets the repaired implementation. BTD is not computed during generation. It is used only for scoring afterward. The experiment therefore compares "repair loop vs. no repair loop" and different proposal strategies. It does not test the effect of using BTD as an objective, and the label is misleading.
    • The evaluation is circular. Expansion is judged only by ΔBTD. There is no independent check such as human semantic labels, mutation score on the reference implementation, or a count of real specification gaps found. There is one qualitative example (relative paths).
    • Statistical support is weak. Each condition has 3 runs. The no-repair-loop baseline varies widely (2.945, 1.600, 0; s² = 2.17). The comparison of 3.857 (Reach/Kill) vs. 1.515 has no test. The Coverage variant's budget is not stated in the main text, and the reason it adds 0 tests is not explained.
    • Budgets differ. Repair-loop methods run a repair after each accepted test, and this agent cost is not counted in the "24 attempts." The baseline has no repair step.
    • Fix: rename the method (for example "repair-loop expansion") or add a variant that selects by ΔBTD and compare against it. Add an evaluation independent of BTD, use at least 5–10 runs with confidence intervals, and count repair cost in the budget.
  4. BTD lacks external validation against real behavioral differences. BTD measures whether one agent's repair for one test also passes another test. This differs from two tests constraining the same behavior.

    • Repair granularity changes BTD directly. An agent that rewrites broadly or fixes several things at once raises transfer and lowers BTD. An agent that makes minimal edits raises BTD. BTD thus mixes test properties and repair style. Sec. 3.3 and 5.4 read a lower BTD as higher "alignment efficiency," but more aggressive repair gives the same number. The paper does not separate these sources.
    • There is no comparison with human labels or an external reference. The only external checks are 3 hand-picked triples in Fig. 7 and the 13-test group in sd. I suggest labeling behavior classes on several tasks (by humans or by specification clauses), reporting agreement with BTD groups (e.g., ARI), and comparing with simple proxies such as coverage similarity, output uniqueness, and mutant-kill-vector similarity.
    • The test subsets are constructed. App. D.2 says each 100-test subset includes "nine source-reviewed anchors with related local checks," so related tests were placed in on purpose. The Sec. 4.3 claim of uneven redundancy in existing suites therefore partly reflects subset construction. Only sd (App. E) uses the full official suite.
  5. Design choices are not ablated, and the estimator is biased with no uncertainty.

    • Symmetrization. Eq. 3 uses min(P_ij, P_ji). Fig. 7 and Fig. 10 both show clear one-way transfer. For pruning, one-way inclusion is the most useful information, since keeping the stronger test covers the weaker one. The min discards it. Please compare with max, mean, and selection based on directed inclusion.
    • Failing tests only. BTD uses only T, so tests that C0 passes never enter the measure. BTD measures the diversity of C0's remaining defects, and "test suite diversity" in the title and abstract is too broad. Different C0 give different T (n = 30, 45, 36 in Table 3), so these BTD values are not strictly comparable.
    • Small-R bias. With R = 2, P̂ takes only 0, 0.5, or 1. Taking the min of two noisy estimates biases S downward and BTD upward. The paper does not analyze this and gives no bootstrap intervals. Tables 3 and 4 compare at R = 2.
    • Repair failures. Algorithm 2 returns ⊥ (BTD undefined) if any repair fails. Unrepairable tests are common in practice. The paper does not report how often this happens or whether tasks or tests were excluded, which may cause selection bias.
  6. The agent comparison (Sec. 5.4, Table 3) claims more than the evidence supports.

    • The three agents differ in C0, failing set, and repairer, so BTD mixes documentation understanding and repair style (see item 4).
    • The R = 1 scores in App. D.5 are GLM 25 / GPT 30 / Opus 26 (replicate 0) and 26 / 28 / 26 (replicate 1). The GLM vs. Opus difference (25.3 vs. 26.0) is within replicate variation. There is one task (bedtools2), one C0 per agent, and R = 2.
    • The paper states that BTD gives the same ranking as pass rate, so the claim that BTD complements pass rate is unsupported.
    • Fix: compare on several tasks, fix the repairer and vary only the C0 author (or the reverse), and report confidence intervals.
  7. Data contamination is not discussed. sd, bedtools2, lz4, jsonschema, yj, and trdsql in ProgramBench are public open-source projects, and vLLM's block_pool.py is widely seen. GLM-5.2, GPT-5.5, and Opus 4.6 have likely seen this code. C0 written "from documentation only" and later repairs may rely on memorized reference code. This affects two results. Repair transfer may reflect convergence toward a memorized implementation instead of relations between tests. Full recovery after pruning may also partly come from memory. I suggest adding projects released after the models' training cutoffs, or controls with obfuscated identifiers and interfaces, and at least discussing this in the Limitations.

  8. The ComputeEval experiment (Sec. 5.3, Table 2) barely tests BTD. For CUDA/104 and CUDA/197, S is all ones and BTD = 1, so any single test is equivalent and every selection method gives the same result. LayerNorm has BTD = 1.3, involving only two entries below 1. The three tasks were chosen because the documentation-based implementation happened to fail, and they are small. This shows that the protocol runs on GPU kernels. It does not support the claim that "BTD generalizes across software domains." Tasks with more failing tests and richer structure, plus a random baseline, would help.

Minor issues

  1. Table 2 reports "30/30" recovery for CUDA/104, while Table 7 and App. F.2 report "3/3" and three repairs per single-test selection. Please reconcile these and explain where 30 comes from.
  2. The notation BTD(T′; S[T′,T′]) in Eq. 7 differs from BTD(T | C0, d, protocol, R) in Eq. 4. The pruning search algorithm (greedy or enumeration, and how 186 sd tests are handled) is not given in the main text. App. F.2 describes only the ComputeEval enumeration.
  3. "C1 C2 C3" in Fig. 3(c) and "Candidate 1, 2, 3 correspond to the earlier C0 and later C1 and C2" in App. D.2 conflict with the repair-candidate notation C_{i,r} in Sec. 3.1.
  4. "Alignment efficiency" in Sec. 3.3 is descriptive and not formalized.
  5. App. A proves only that BTD does not decrease under extension. The Sec. 3.2 claim that adding a fully overlapping test leaves BTD unchanged needs a condition (identical overlap with all other tests). Please state and prove it in the appendix.
  6. Related work is missing: fault-detection-preserving test suite reduction beyond Yoo & Harman; test diversity measures (Feldt et al., test set diameter / NCD); output uniqueness (Alshahwan & Harman 2014); mutant subsumption and minimal mutant sets (Ammann et al.; Kurtz et al.), which also use dynamic inclusion to find redundancy and are conceptually closest; and test-equivalence analysis in program repair (Mechtaev et al., 2018).
Questions for authors
  1. On sd, can random subsets of 72 failing tests recover 810/810? Please report recovery rates over several seeds.
  2. On vLLM, how does "one random test per connected component" compare with the BTD selection? What are the results for both methods at budgets 5, 6, and 8?
  3. How many tokens, how much money, and how much time did the 186 × 186 sd matrix (744 repair trajectories) cost? How does this compare with repairing directly against all 810 tests?
  4. Do the "BTD-guided" expansion methods compute or use BTD during generation? If not, why this name? What is the Coverage variant's budget, and why did it add 0 tests?
  5. Across all experiments, how many repair trajectories failed to pass the target test within the deadline? Did this lead to excluding any tasks or tests?
  6. Why use min instead of keeping the directed P? Would pruning based on directed inclusion work better?
  7. How do you rule out memorization of sd, bedtools2, and vLLM source code affecting C0 and repair transfer?
  8. How much does BTD change on the same test set when the repair agent changes? If C0 is fixed and only the repair agent is swapped, how does BTD move?
  9. What do "30/30" in Table 2 and "3/3" in Table 7 refer to for CUDA/104? How many failing tests does each ComputeEval task have?

Suggestions (not affecting my score)

  • State the full source of "61–90%" (n and number selected per task) in the main text.
  • Simplify Fig. 1 (left) or explain how the PR numbers "estimated from Haque et al. (2026, Table 1)" and labels such as "~128" were obtained.
  • Since results depend on specific providers (OpenRouter, Z.AI FP8) and subscription clients, please share anonymized repair logs and S matrices during review.
5 comments

We sincerely thank you for your exceptionally detailed and insightful feedback. We greatly appreciate the time and effort you devoted to carefully examining our work. Your comments have been extremely valuable in helping us identify important areas for improvement and substantially strengthen both our methodology and experimental evaluation. Motivated by your suggestions, we conducted several additional experiments. Below, we present our response so far to several of the points raised.

W1 / Q1: Strengthen pruning baselines and statistical reliability.

We agree with the importance of controlled baseline comparisons. We added experiments comparing BTD against random, branch-coverage-guided, and mutation-guided test selection on the full sd task. Coverage is measured using LLVM-based instrumentation, while mutation-based selection uses cargo-mutants. All methods select 72 failing tests, retain the same 624 regression tests, and use the same downstream repair procedure.

Score ( sample size 5)
Random794.20±6.50794.20 \pm 6.50
Coverage792.80±6.98792.80 \pm 6.98
Mutation795.60±5.55795.60 \pm 5.55
BTD809.20±1.79809.20 \pm 1.79

Post-hoc analysis shows that these gains include concrete requirements such as non-UTF-8 input handling and duplicate-option validation. Some requirements were indistinguishable under the measured coverage and mutation features but were not interchangeable under observed repairs. These cases illustrate the behavioral significance of the aggregate improvement.

To further strengthen the statistical evidence, we conducted 10 additional independent repair runs using the BTD-selected 72-test subset. The resulting implementations passed 808.50±2.72808.50 \pm 2.72 out of 810 tests, with complete recovery in 7/10 runs.

These additional experiments demonstrate that BTD-selected tests provide more effective corrective guidance than conventional test-selection criteria under the same feedback budget.

Q2: Recovery across different test budgets and the contribution of MILP aggregation.

Thank you for suggesting a more systematic evaluation of selection budgets and the contribution of MILP aggregation. We report more data of our vLLM experiments to compare BTD and coverage-based selection across multiple test budgets, using the same initial implementation ((C0C_0): 53/81 passing tests) and downstream repair protocol.

Recovery across selection budgets. We evaluated both methods during subset search and through independent repairs using frozen selections.

Sample size 3BTDCoverage
K=5K=578.67±0.5878.67 \pm 0.5875.00±0.0075.00 \pm 0.00
K=6K=680.33±1.1580.33 \pm 1.1575.67±2.0875.67 \pm 2.08
K=7K=781.00±0.0081.00 \pm 0.0078.00±1.7378.00 \pm 1.73
K=8K=881.00±0.0081.00 \pm 0.0081.00±0.0081.00 \pm 0.00

These results show that BTD's advantage is most pronounced under constrained feedback budgets. We will include the budget-dependent recovery results and clarify the distinction between subset search and fixed-selection evaluation in the revised paper.

Contribution of MILP aggregation. We also evaluated whether selecting representatives independently within behavioral blocks can replace BTD's global aggregation. On sd, we compared full BTD selection with two block-wise alternatives, keeping the same 70-test budget and per-block selection allocations.

Sample size 5Score
BTD with aggregation807.40±3.29807.40 \pm 3.29
BTD(R=1R=1, random select from each block)804.20±6.91804.20 \pm 6.91
Min_transfer_for_test_within_block800.00±11.81800.00 \pm 11.81

When R=1R=1, BTD degenerates to random representative selection within each behavioral block. With additional repair samples, BTD captures fractional repair-transfer relationships, enabling MILP aggregation to make more fine-grained, diversity-aware selections.

Across five independent repair runs, BTD with MILP aggregation achieves higher average recovery than both block-wise alternatives, outperforming random selection by 3.2 tests and minimum-transfer selection by 7.4 tests, with lower variability.

These results suggest that global diversity-aware aggregation can provide benefits beyond simply identifying behavioral groups. We will incorporate this ablation and clarify the respective roles of repair-transfer-based grouping and MILP aggregation in the revised paper.

W2 / Q3: Clarify the reduction scope and computational cost.

We appreciate your suggestion to clarify the scope of test reduction. Our reported 61–90% reduction refers specifically to initially failing tests, which serve as additional executable specifications for the coding agent.

For sd, BTD reduces the failing tests from 186 to 72 (61.3%). Including the 624 initially passing regression tests, the total number of checks decreases from 810 to 696 (14.1%). We will explicitly distinguish these two measurements.

Regarding computational overhead, each profiling round on sd requires 186 targeted repair trajectories. The full R=4R=4 measurement involves 744 repair trajectories and cross-test evaluations. Using GLM-5.2 (Z.AI via OpenRouter) with the Codex harness, the recorded measurement cost is approximately $132.33 for each round**, $529.33** in total. We will report the computational costs of experiments throughout the evaluation section in the revised paper.

We also measured the cost of jointly repairing C0C_0 using the 72 selected failing tests. We report this downstream repair cost separately from the upfront profiling cost. The repair cost is $7.24.

To improve computational efficiency, we additionally developed UniBTD, a lightweight proxy for BTD based on unidirectional repair transfer. Starting from the initial implementation, UniBTD repairs a selected failing test and groups other tests that are also satisfied by the repaired candidate. Repeating this procedure over the remaining tests approximates behavioral redundancy without requiring the full bidirectional transfer matrix.

Under the same 70-test selection budget, we obtained:

Sample size 5Score
BTD807.40±3.29807.40 \pm 3.29
UniBTD804.00±2.92804.00 \pm 2.92

UniBTD reduces the measured profiling cost by 77.6% (measured by repair count reduction), while achieving comparable downstream repair performance. These results demonstrate that our repair-transfer-based framework enables both the full BTD metric and a lightweight unidirectional approximation, offering a practical trade-off between measurement fidelity and computational overhead.

W3 / Q4: Isolate BTD's contribution to test-suite expansion, and justify reduce human review burden.

Thank you for highlighting the distinction between the repair loop and BTD-based selection. We will clarify that the original expansion experiment primarily evaluates a repair-loop-based generation strategy, with BTD used to measure the resulting diversity.

In the original expansion experiment, BTD was computed after generation to evaluate the resulting suites, rather than used explicitly to select each new test. The name reflected the repair-transfer principle motivating the loop. We will rename these variants “repair-loop expansion” and distinguish them from explicit BTD-based selection in the revised paper.

Regarding the Coverage variant, it searches for inputs that increase branch coverage. No new test was accepted because the candidate and the oracle produced identical outputs for every proposed input, so none exposed a new behavioral discrepancy. We will clarify the acceptance behavior in the revised paper.

To directly isolate BTD's contribution, we added a new controlled experiment in which BTD explicitly guides test selection, while input generation and downstream repair remain identical.

Specifically, an agent first randomly proposes inputs and predicts their expected outputs from documentation. We compare these predictions against the current implementation, retaining inputs that reveal potential behavioral misalignment. We then compare BTD-based clustering and selection against random selection, using the same budget of KK tests. Only the selected tests are queried against the reference implementation to obtain verified outputs, which subsequently guide code repair.

In our largest experiment, the agent proposed 500 inputs, yielding 79 potential misalignment cases (BTD = 39). Across three independent runs, the full-suite recovery results were:

Each with 3 samplesK=2K=2K=6K=6K=10K=10K=20K=20
BTD642.67±10.97642.67 \pm 10.97661.33±14.15661.33 \pm 14.15664.00±15.52664.00 \pm 15.52691.33±20.03691.33 \pm 20.03
Random642.67±11.68642.67 \pm 11.68643.00±14.00643.00 \pm 14.00657.67±18.50657.67 \pm 18.50680.00±9.17680.00 \pm 9.17

BTD-based selection matches or outperforms random selection across all evaluated verification budgets, independently of the input-generation process and the shared repair workflow. At K=6K=6, BTD recovers 18.33 more tests on average than random selection under the same verification budget. Moreover, BTD with only 6 verified tests achieves higher average recovery than random selection with 10 verified tests (661.33 vs. 657.67), requiring 40% fewer verifications. These results demonstrate that BTD-based clustering helps prioritize more informative behavioral specifications, achieving better repair outcomes with the same verification effort, or comparable outcomes with fewer verifications.

Since verifying expected outputs represents a key component of human test-review effort, these findings also highlight BTD's potential to reduce human review burden while preserving effective corrective guidance. We will incorporate this controlled comparison and clarify BTD's explicit role in test selection in the revised paper.

Q5: Repair failures and potential selection bias.

In our benchmark evaluations, all targeted repairs ultimately met the acceptance criteria. The few initially unsuccessful attempts were primarily caused by insufficient agent context rather than unrepairable tests. For example, increasing the context window from 16K to 32K tokens allowed previously unsuccessful repairs to complete successfully. Thus, these context-window limitations did not require excluding any tasks or tests from our evaluation. We will specify the context-window settings, retry procedure, and task/test inclusion criteria in the revised experimental protocol.

Comments on the paper

2 comments

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful and constructive feedback. Your comments have helped us identify important areas for improvement and motivated us to conduct additional experiments to strengthen our methodology, evaluation, and practical applicability.

As the discussion period concludes, we would like to summarize how we addressed the reviewers' key concerns here:

1. Does BTD provide more effective guidance for test-suite pruning?

We strengthened our pruning evaluation through two complementary experiments:

  • Comparison with alternative selection criteria. On the full sd task, we added head-to-head comparisons against random, branch-coverage-guided, and mutation-guided selection, using the same 72-test budget and downstream repair procedure. Across five independent runs, BTD achieved 809.20/810 passing tests on average, compared with 794.20, 792.80, and 795.60 for the respective baselines.

  • Recovery across selection budgets. We extended our vLLM evaluation to compare BTD and coverage-based selection across budgets of 5–8 tests. BTD achieved full recovery (81/81) with 7 selected tests, whereas coverage-based selection required 8, demonstrating BTD's advantage under constrained feedback budgets.

2. Does BTD itself contribute to test-suite expansion?

We clarified that the original expansion experiments used BTD to evaluate the resulting test suites, rather than explicitly guiding test generation. We will distinguish these repair-loop-based experiments from explicit BTD-guided selection in the revised paper.

To directly isolate BTD's contribution, we added a controlled experiment in which BTD guides test selection from a shared pool of candidate inputs, while keeping input generation and downstream repair identical to random selection. Only selected tests are verified against the reference implementation.

With a budget of 6 verified tests, BTD achieved 661.33 passing tests on average, compared with 643.00 for random selection. Moreover, BTD with 6 verified tests outperformed random selection with 10 (661.33 vs. 657.67), requiring 40% fewer verifications. This provides direct evidence that BTD can prioritize more informative behavioral specifications independently of the shared generation and repair workflow.

3. How reliable are the metric's design choices and estimates?

  • Mutual repair transfer. We clarified that the use of min⁡\min symmetrization is a conservative design choice intended to identify mutually substitutable repair guidance. We added min⁡/mean/max⁡\min/\mathrm{mean}/\max ablations and stress tests using unfavorable test selections. Under these conditions, min⁡\min achieved 798.20 passing tests on average, compared with 783.80 for mean and 766.60 for max, supporting its robustness for downstream repair.

  • Contribution of global aggregation. On sd, we compared full BTD selection against two block-wise representative selection strategies under the same 70-test budget. BTD with aggregation achieved 807.40 passing tests on average, compared with 804.20 for random within-block selection and 800.00 for minimum-transfer selection. These results suggest benefits from global diversity-aware MILP aggregation beyond behavioral grouping alone.

  • Sampling uncertainty. We extended the analysis to six repair trajectories per target and evaluated BTD estimates across sampling budgets. The results characterize finite-sample inflation and show decreasing variability with larger sampling budgets. We also performed 10 independent downstream repair runs using the selected 72-test subset, achieving 808.50±2.72808.50 \pm 2.72 passing tests out of 810, with complete recovery in 7/10 runs.

These experiments provide additional evidence for the effectiveness of our design choices.

4. What are the practical benefits and computational costs?

We clarified the scope of BTD's benefits and explicitly quantified its overhead.

  • Scope of reduction. The reported 61–90% reduction concerns initially failing tests used as additional executable specifications. On sd, BTD reduces these tests from 186 to 72 (61.3%), while the complete feedback suite decreases from 810 to 696 tests (14.1%), including initially passing regression tests.

  • Computational overhead. We measured BTD profiling costs and separated them from downstream repair costs. On sd, full profiling with R=4R=4 requires 744 repair trajectories, costing approximately $529.33.

  • Lower-cost approximation. To address profiling overhead, we developed UniBTD, which approximates behavioral redundancy using unidirectional repair transfer. UniBTD reduces profiling repair counts by 77.6% (measured by test count) while achieving comparable downstream repair performance.

We also clarified the intended practical workflow: BTD aims to reduce redundant test-as-spec feedback and the number of behavioral specifications requiring verification or human inspection, rather than simply minimizing the total number of regression checks.

Additional methodological clarifications

We addressed the reviewers' questions regarding repair failures and retry procedures, reconciled the ComputeEval test-count reporting, and acknowledged limitations including potential pretraining exposure to public repositories. We also clarified the scope of our evaluation on documentation-driven agentic code generation and repair using real-world software benchmarks.

We believe these additional experiments and clarifications substantially strengthen the empirical evidence for BTD, more clearly isolate its contributions, and provide a more transparent characterization of its practical benefits and limitations. We will incorporate these results and clarifications into the revised paper.

Thank you again for your time, careful consideration, and valuable feedback throughout the review process.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.