acceptodds
Under review as a conference paper at ICLR 2027

BehaviorDistill: Distilling Black-Box Software via Evolving, Verifiable Specifications

Abstract

Black-box software replication builds a student program from scratch to match a teacher's behavior using documentation and execution observations without source access. Recent methods capture observed behavior in specifications to guide construction, but natural-language specifications cannot be executed to verify consistency or locate discrepancies. Tests enable such verification, yet handcrafted generation rules restrict exploration and risk biased, incomplete coverage, while unstructured suites obscure behavioral relations and coverage gaps. We introduce BehaviorDistill, a method connecting exploration and construction through evolving, verifiable behavioral specifications. Evolving Class-Structured Coverage (ECC) synthesizes these specifications by linking executable tests to an evolving class-structured behavior model, where high-level classes guide feature discovery, fine-grained classes guide deeper exploration, and new tests refine the model. Building on these specifications, Plan-Guided Coverage-Expanding Construction (PCC) uses dynamic plans to organize test constraints into staged implementation and repair tasks, while construction feedback drives test expansion and revisions to plans and code. Across multiple benchmarks and models, BehaviorDistill achieves Raw Pass Rate and Almost Resolved gains reaching 15.8 and 18.2 percentage points, respectively, illuminating a path toward autonomous software reconstruction through evolving, verifiable behavioral specifications.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Black-Box Software Replication, LLM Agents, Behavioral Specification, Automated Test Generation, Hierarchical Exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HokBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the BehaviorDistill framework. It employs "Evolving Class-Structured Coverage" (ECC) to construct structured behavioral specifications that evolve continuously throughout the testing process. Subsequently, it utilizes "Plan-Guided Coverage-Expanding Construction" (PCC) to organize the implementation of tests and the repair of code based on these specifications, thereby enabling the behavioral model, the test suite, and the software implementation to co-evolve throughout the exploration process. The problem holds practical significance; the proposed method bridges "exploration" and "construction," and its effectiveness has been verified through experiments.

Critical strengths
  1. A black-box software specification method is proposed that integrates executable tests with structured behavioral models. This approach allows specifications to not only guide development but also facilitate verification, thereby enabling the automated testing and repair of software code.
  2. BehaviorDistill introduces two processes: ECC (Explore and Construct Structured Behavioral Specifications) and PCC (Implement based on these specifications, while continuing to uncover omissions during implementation), to automatically explore the behavior of a black-box program and build, from scratch, a "student" program that exhibits similar behavior.
Critical weaknesses

The problem definition and formulation require further refinement. For instance, what are the specific scenarios or types of software for which the proposed method is intended? Additionally, what is the rationale behind the class structure/behavioral models and behavioral specifications? Could you provide a more detailed explanation of your approach?

Questions for authors
  1. How is the construction of behavioral models and ECC coverage managed to ensure that it does not negatively impact the performance of students' subsequent programs? Is there a performance-related correlation between the two?
  2. The paper mentions "scenarios where software source code is not accessed"。In such cases, does the source code have any bearing on the exploration of ECC or the construction of PCC?
7 comments
AuthorsOpenReview ↗

Response to Reviewer HokB (1/4)

We thank the reviewer for the valuable comments and for recognizing the practical significance and method design of our work. The target application scenario, motivation for behavioral specifications and the class-structured behavior model, synergy between the two stages, and strictly source-free black-box setting were all explicitly stated and empirically validated in the Abstract, Sections 1–3, and Appendices C.1–C.4 of the original submission. Below, we clarify each point with explicit references to the submitted manuscript:

[Paper Summary, Critical Strengths & W1] Target Scenario and Software Types, Motivation for Behavioral Specifications and Class-Structured Behavior Models, and Method Overview

  1. The target scenario and applicable software types were explicitly defined in the Abstract, Section 2.1, the opening paragraph of Section 2.2.1, and Appendix C.1 of the original submission: We focus on black-box software replication: given a target black-box program with no source code and only documentation and an executable binary as the teacher, the goal is to build a behaviorally consistent program from scratch as the student and evaluate it on hidden tests; behavioral specifications model the target black-box software's behavior and functionality to guide replication from scratch, rather than testing or repairing an existing source repository (Abstract; Sec. 2.1). Our method targets standalone deterministic programs invoked via command-line scripts whose behavior is observed through standard output, standard error, exit status, and file artifacts (Secs. 2.1, 2.2.1). Represented by ProgramBench and MirrorCode, the tasks span small command-line utilities to large complex software projects, covering command-line and Unix utilities, data processing tools, databases, interpreters, static analyzers, compression software, and multimedia software (Appendix C.1).
AuthorsOpenReview ↗

Response to Reviewer HokB (2/4)

  1. The design motivation for behavioral specifications and the class-structured behavior model was systematically presented in the Abstract, Section 1 (Paragraphs 2–3), and Section 2.2.1 (Paragraphs 1–3) of the original submission:
    • Software documentation is often coarse-grained and leaves detailed feature interactions and boundary cases unspecified, requiring agents to actively probe the black-box teacher and abstract its behavior into a specification to guide replication. Prior approaches such as SpecFirst rely on natural-language specifications, which cannot be executed to verify behavioral consistency or pinpoint discrepancies; behavioral specifications must therefore include executable tests (Abstract; Sec. 1).
    • However, while unstructured collections of tests verify individual observed behaviors, they obscure behavioral subspaces and their relations within a vast behavior space; a behavior model is therefore needed to organize tests structurally, locate coverage gaps, and balance exploration breadth and depth under a finite budget (Abstract; Secs. 1, 2.2.1).
    • Conventional behavior models such as state machines and input classification trees typically focus on a single behavioral dimension, rely on manual abstraction, and lack a unified representation for direct composition and refinement. We therefore represent the class-structured behavior model as a code skeleton containing only class and function declarations: classes represent behavior subspaces, functions represent operations, and inheritance, ownership, reference, and call edges form a typed multirelational graph that losslessly unifies and composes state machines, input classification trees, and dependency graphs; each test is uniquely assigned to one class and linked to relevant classes or functions, forming the behavioral specification together with the model (Sec. 2.2.1; Supplementary Formal Proofs, Theorem 2.2, Prop. 2.3).
  2. The algorithmic workflow and co-evolving artifacts of ECC and PCC were detailed in Figure 1, Section 1 (Paragraphs 4–6), and Sections 2.2–2.3 of the original submission:
    • ECC is the coverage-driven exploration stage, in which the behavior model and the test suite co-evolve while probing the teacher (Sec. 2.2.2). Exploration Agents first query the teacher to generate initial tests and synthesize an initial class structure, then use the class structure as a coverage map to explore across levels in parallel: high-level classes discover uncovered major features, while fine-grained subclasses probe input boundaries, state changes, and feature interactions. New tests are assigned to classes via the lowest common ancestor (LCA) rule and trigger subclass splitting when a class exceeds the load limit, without relying on handcrafted test dimensions or combination rules as in Gongent (Secs. 1, 2.2.2).
    • PCC is the student construction stage, during which the ECC model and test suite remain fixed, while the dynamic plan, the Builder's self-generated test suite, and the student code co-evolve (Sec. 2.3). Guided by the behavior model and test scripts, the Builder uses a dynamic plan to organize test constraints into staged student implementation and repair tasks, then iteratively implements or repairs code according to the plan, probes the teacher on uncovered areas to expand its self-generated test suite, executes both test suites, and revises the plan from test feedback until student construction is complete (Sec. 2.3). Using black-box exploration alone, our method reaches 51.1% test coverage on MirrorCode—surpassing the human-revised official suite (47.1%)—and improves Raw and AR over Vanilla by up to 15.8 and 18.2 percentage points (Table 1).
AuthorsOpenReview ↗

Response to Reviewer HokB (3/4)

[Q1] How Are Behavior Model Construction and ECC Coverage Managed to Avoid Negative Effects, and Do They Correlate with Student Performance?

  1. Sections 2.2.2, 2.3, and 3.4 and Appendix Table 4 of the original submission explicitly detailed three mechanisms that prevent specifications from negatively affecting student construction:
    • Every test in ECC must be executed on the real teacher with its actual output captured before entering the suite, and the class-structured behavior model serves solely as a behavioral coverage map without constraining the internal architecture of the student implementation (Secs. 2.2.2, 2.3; Supplement 03).
    • PCC separates the fixed ECC suite into input-only test scripts and hidden expected outputs stored in a validation environment; the Builder can inspect only the input scripts and receives pass/fail feedback through test execution, preventing hardcoded outputs (Sec. 2.3).
    • Handing a large suite of fine-grained tests all at once to an unplanned builder can indeed overwhelm it—for instance, in Table 4, ECC + Vanilla sees Hard Raw drop from Vanilla's 45.3% to 41.8%. PCC resolves this by using a dynamic plan to decompose large test suites into staged implementation and repair tasks, lifting Hard Raw to 56.7% and overall performance to 77.5% Raw and 32.0% AR (Secs. 2.3, 3.4; Appendix Table 4).
  2. Section 3.2 (Table 1) and Section 3.4 (Appendix Tables 4 and 5) of the original submission, together with our supplementary experiments, empirically demonstrate that ECC coverage correlates positively with student replication performance and translates fully into gains when paired with PCC:
    • Across all four experimental settings in Table 1 of the original submission, BehaviorDistill achieves substantially higher test coverage than Gongent alongside higher Raw and AR across the board (Sec. 3.2; Table 1). Moreover, when the three exploration variants in Appendix Table 5—Flat, Tree-only, and Full ECC (whose exploration coverage increases from 58.5% to 63.9% to 69.8%)—are paired with the identical PCC stage, downstream Raw/AR rises monotonically from 71.7%/27.5% to 73.8%/29.0% to 77.5%/32.0% (see the supplementary experiment in our response to Reviewer U7xW [W1 & Q1]), and across the 200 tasks of ProgramBench (DeepSeek-V4-Flash-0731), ECC test coverage exhibits a Spearman rank correlation of 0.64 with hidden-test pass rate.
    • At the same time, high coverage alone is not sufficient: in Table 4, ECC + Vanilla achieves higher coverage (69.8%) than PCC only (62.6%) but lower Raw and AR, demonstrating that high-coverage specifications must be paired with PCC's staged planning mechanism to translate effectively into student replication performance (Sec. 3.4; Appendix Table 4).
Reviewer_U7xWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BehaviorDistill, a black-box software replication framework. It uses a class-structured model, which is a code skeleton of class and function declarations, to express behavior relations and to locate coverage gaps in the test set.

In the ECC module, this model represents the teacher’s behavior and hierarchical exploration to generate new tests. In the PCC module, a builder agent constructs the student program from these tests and observations. As a result, BehaviorDistill provides a more thorough test set than existing baselines. This raises test coverage and improves the student’s replication performance, especially Almost Resolved (AR) score.

Critical strengths

S1. A reasonable framework for solving black-box software replication BehaviorDistill organizes the teacher’s behavior with a hierarchical code skeleton. With this structure, 1) it covers diverse behaviors horizontally with high-level classes, and 2) it covers detailed behaviors vertically through deeper exploration at fine-grained classes. This design seems useful and reasonable. It also seems to be a key to addressing the difficulty of exploring the teacher’s behavior to build a test set in a software replication task.

S2. Consistent improvements in coverage and AR Using this code model, BehaviorDistill reaches higher line coverage than the official tests of MirrorCode. This shows that the generated test set has good quality and high coverage. In addition, AR is consistently higher than the baselines. This shows that the test set covers the task thoroughly.

Critical weaknesses

W1. The replication gain from the class-structured is unclear The main contribution is the class-structured specification, and the idea seems reasonable. However, no experiment clearly separated its effect on replication performance. Table 5 shows the structural ablation, but it only reports coverage, not replication. This matters because higher coverage does not lead to better replication. For example, in Table 4, “ECC + Vanilla" has higher coverage than “PCC only” but PCC only has higher Raw and AR.

W2. Fairness of the comparison with baselines It is also unclear whether BehaviorDistill is compared with baselines under fair settings. For example, the prompt level (including the finish rule) and the total budget differ, and the performance gap might come from these differences. This matters because the gap becomes much smaller under more equal settings, such as under the continuation protocol in Table 3.

Questions for authors
  1. Table 5 shows that the class structure improves coverage, but it does not report replication performance. Could you report Raw and AR for each variant in Table 5, each followed by PCC?

  2. The LLM prompts in the supplementary are well designed, while Vanilla might use the default mini-SWE-agent prompt. For example, the Builder prompt allows “finish” only when several conditions hold, but the baselines might not have such a rule. Under the continuation protocol in Table 3, the 6-hour baselines improve a lot. How much of the performance gain comes from the richness of the prompts?

  3. The total budget is also unequal for Vanilla. BehaviorDistill uses 6+6 hours, while Vanilla uses only 6 hours. Within 12 hours, Vanilla reaches 77.8 on Raw. In addition, ECC runs 3 agents per class in parallel, so equal wall-clock time does not mean equal computation. Could you report the actual tokens and time used by each method, and does BehaviorDistill still outperform the baselines under the same total budget?

  4. The paper states that previous works often rely on human priors. However, the prompts of BehaviorDistill also seem to contain some human priors, such as criteria for refining subspaces and fixed roles for each class level. In addition, most prompt examples are about database CLIs. How is BehaviorDistill different from previous works in this respect?

15 comments
AuthorsOpenReview ↗

Response to Reviewer U7xW (1/6)

We thank the reviewer for the constructive feedback and for recognizing our hierarchical code-skeleton design, test suite quality, and improvements in source-code coverage and AR. Before responding to your specific questions, we clarify two key mechanisms related to the summary and strengths:

  1. ECC's class-structured behavior model is not a simple hierarchical tree, but a typed multirelational graph with four edge types—inheritance, ownership, reference, and call—that organizes behavioral hierarchy through high-level classes and fine-grained subclasses while capturing cross-module dependencies and call interactions. Moreover, the model is not static: as new tests are assigned via the lowest common ancestor (LCA) rule and classes exceeding the load limit automatically split into subclasses, the model co-evolves with the test suite (Secs. 2.2.1–2.2.2; Table 5).
  2. PCC does not write code in a single pass from the fixed ECC test suite; instead, it uses a dynamic plan to decompose the behavior model and test scripts with expected outputs removed into staged tasks while continuing to probe the teacher during construction to expand its self-generated test suite, co-evolving the dynamic plan, self-generated test suite, and student code (Sec. 2.3; Tables 4, 6). Consequently, across all four settings in Table 1, our method substantially improves teacher line coverage alongside Raw and AR, boosting the latter two by up to 15.8 and 18.2 percentage points over Vanilla (Sec. 3.2; Table 1).
AuthorsOpenReview ↗

Response to Reviewer U7xW (2/6)

[W1 & Q1] Downstream PCC Results for Table 5 Structural Variants and Analysis of ECC + PCC Synergy

  1. Following your suggestion, we connected each of the three exploration variants from Appendix Table 5 to the identical PCC stage on ProgramBench, yielding the results below:
Exploration Variant (followed by identical PCC)Exploration Cov. (%)Post-PCC Cov. (%)Post-PCC Raw (%)Post-PCC AR (%)
Flat + PCC58.563.771.727.5
Tree-only + PCC63.967.673.829.0
Full ECC + PCC69.871.377.532.0
  1. Multirelational class-structured specifications substantially raise exploration coverage and the replication ceiling, while high-coverage specifications must be paired with PCC's staged planning and test expansion to translate fully into replication gains (Sec. 3.4; Appendix Tables 4, 5):
    • Full ECC's hierarchical and multirelational structure is key to improving both coverage and replication performance. First, compared with unstructured Flat + PCC, Tree-only + PCC raises exploration coverage from 58.5% to 63.9% and Raw/AR from 71.7%/27.5% to 73.8%/29.0%, whereas Flat + PCC performs similarly to PCC only without preliminary exploration at 71.6%/28.0%. This indicates that accumulating unstructured tests fails to expose coverage gaps, making class inheritance and function ownership hierarchies essential for coordinating breadth and depth. Second, Full ECC + PCC further improves over Tree-only + PCC by pushing exploration and post-PCC coverage to 69.8% and 71.3% and Raw/AR to 77.5%/32.0%, showing that adding reference and call edges beyond the tree hierarchy captures cross-module dependencies and feature interactions to guide more complete behavioral replication (Secs. 2.2, 3.4; Appendix Tables 4, 5).
    • High-coverage specifications from ECC and plan-guided construction in PCC are mutually indispensable. In Table 4, ECC + Vanilla achieves higher coverage at 69.8% than PCC only at 62.6%, yet its Raw/AR of 69.4%/25.0% is lower than PCC only's 71.6%/28.0% because unplanned Vanilla cannot digest a large suite of fine-grained tests at once, causing Hard Raw to drop from 45.3% under Vanilla alone to 41.8%. Conversely, PCC only builds in stages but is limited to 62.6% coverage without preliminary global exploration. The full ECC + PCC pipeline combines high-coverage structured specifications from ECC with PCC's staged planning and construction-time test expansion to 71.3% coverage, achieving 77.5%/32.0% overall Raw/AR and 56.7% Hard Raw (Secs. 2.3, 3.4; Appendix Table 4).
AuthorsOpenReview ↗

Response to Reviewer U7xW (3/6)

[W2 & Q2] Impact of Prompt Richness and finish Rules on Performance Gains

  1. The finish rules you noted refer to the five completion conditions in the Builder Agent prompt (Supplement 03):

    • "every task-specific completion-checklist item is covered and every TO_EXPLORE item has been resolved with evidence, AND"
    • "the self-generated testcase bank is REASONABLE and COMPLETE for the currently identified scope (you actively tried to find relevant reference behaviors the two suites do not cover, and found no remaining concrete gap), AND"
    • "the fixed ECC validation command shows ALL ECC cases pass, AND"
    • "bash tests/casebank.sh shows ALL cases pass (CASEBANK: N/N), AND"
    • "PLAN.md has no pending tasks."

    Because all five conditions are tied to ECC and PCC artifacts and cannot be directly copied into Vanilla, the most direct comparison under the identical prompt is PCC only, which uses the exact same Builder Agent prompt. Nevertheless, to isolate the effect of stopping conditions on Vanilla, we designed a Vanilla-specific stopping-rule ablation mirroring the five rules above:

    • "every feature and uncertain item in the task documentation and --help output is covered and has been verified with evidence, AND"
    • "your black-box probing is REASONABLE and COMPLETE for the currently identified scope (you actively probed the reference on normal, boundary, and error inputs, and found no remaining concrete gap), AND"
    • "ALL documented examples and usage patterns produce identical outputs on your implementation and the reference, AND"
    • "ALL probed normal, boundary, and error inputs match the reference in stdout, stderr, and exit code, AND"
    • "the current implementation has no unresolved issues, unimplemented features, approximate workarounds, stubs, or parsed-but-ignored flags."
  2. We compare this ablation variant against Vanilla, PCC only, and full BehaviorDistill on ProgramBench:

MethodConfigurationRaw (%)AR (%)
VanillaDefault mini-SWE-agent prompt69.318.5
Vanilla + Custom finish rulesDefault prompt + PCC-mirrored stopping conditions70.520.0
PCC onlyFull Builder Agent prompt, without ECC specification71.628.0
BehaviorDistillFull Builder Agent prompt, ECC + PCC77.532.0
  1. Results show that prompt-level finish rules contribute minimally to performance gains; BehaviorDistill's advantage stems primarily from ECC's class-structured specifications and PCC's dynamic planning and test expansion (Secs. 3.3–3.4; Appendix Tables 3, 4, 6):
    • Merely restricting early finish via prompting yields negligible gains, and prolonging construction without regression tests can even introduce regressions. Adding custom finish rules to Vanilla nudges Raw/AR only from 69.3%/18.5% to 70.5%/20.0%, still far below PCC only (71.6%/28.0%) and BehaviorDistill (77.5%/32.0%). Moreover, when early termination is completely eliminated via 6h forced continuation in Appendix Table 3, Vanilla's Raw rises to 73.5% by fleshing out incomplete tasks, but lacking a persistent regression suite to safeguard working features causes continued edits on near-complete tasks to introduce regressions, reducing its AR from 18.5% to 17.5%—whereas BehaviorDistill, protected by dual test suites, rises to 78.5%/33.5%. Thus, without executable test specifications, discouraging early termination cannot ensure deep behavioral consistency (Sec. 3.3; Appendix Table 3).
    • Component ablations clearly isolate the independent contributions of ECC specifications, dynamic planning, and test expansion. First, compared with PCC only without preliminary specifications (71.6%/28.0%), adding ECC class-structured specifications boosts BehaviorDistill's Raw/AR to 77.5%/32.0%. Second, in Appendix Table 6, removing dynamic planning or construction-time test expansion from full BehaviorDistill (77.5%/32.0%) reduces Raw/AR to 75.5%/28.0% and 72.4%/21.0%, respectively, directly confirming that our core gains come from the algorithmic mechanisms rather than prompt stopping rules (Sec. 3.4; Appendix Tables 4, 6).
Reviewer_EutdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BehaviorDistill, a framework for replicating black-box software through evolving, executable behavioral specifications. ECC organizes tests into a class structure to guide further exploration, while PCC combines planning, test expansion, and code repair to construct the implementation. The evaluation is fairly comprehensive, and the results are convincing. However, I have concerns about the method presentation and computational-cost comparisons.

Critical strengths
  1. The evaluation is comprehensive, covering two benchmarks and two models (Table 1), scaling experiments (Figures 2–3), ablation studies (Tables 4–6), and results across three runs (Table 8).

  2. The method effectively connects behavioral exploration with code repair. In Table 6, removing test expansion during construction reduces Raw from 77.5% to 72.4% and AR from 32.0% to 21.0%, supporting the value of continuing to discover new behaviors during construction.

Critical weaknesses
  1. The method section is overly complicated. Section 2.2 introduces extensive notation, particularly the rules for test assignment, class loads, and structural updates in Section 2.2.2, steps B–D. These details make the presentation difficult to follow. The authors should explain why these mechanisms are necessary and use a running example throughout the method section to illustrate the concrete problems they address.

  2. The computational-cost comparison requires further clarification. ECC runs exploration agents across multiple classes in parallel, so equal time budgets do not imply equal computational costs. The shared 200M-token experiment in Table 7 is helpful: BehaviorDistill achieves Raw/AR of 78.2%/32.0%, compared with 73.3%/27.5% for SpecFirst. However, actual token consumption for the main experiments in Table 1 should also be reported. Moreover, the construction times in Figure 3 and Table 3 exclude preliminary exploration. An end-to-end cost comparison would better support the efficiency claims.

  3. The necessity of some structural mechanisms is insufficiently justified. In Table 5, Flat, Tree-only, and Full ECC achieve line coverage of 58.5%, 63.9%, and 69.8%, respectively. These results support structured exploration overall, but do not separately establish the necessity of the lowest-common-ancestor assignment rule or class-load policy. Matching the test-count stopping threshold does not establish comparable computational budgets. A comparison with a simpler hierarchical grouping approach under matched token budgets, including downstream Raw/AR results, would help clarify their value.

Questions for authors

If the authors can address the following concerns, I would consider raising my score:

  1. Can you simplify Section 2.2 and use a running example to explain the rationale behind test assignment, class loads, and structural updates?

  2. Beyond Table 7’s fixed-token comparison, can you report actual token consumption for Table 1 and include preliminary exploration costs in the efficiency analysis in Figure 3 and Table 3?

  3. Beyond Table 5’s structural ablation, can you establish whether the LCA assignment and class-load policy are necessary by comparing against simpler hierarchical grouping under matched token budgets, reporting both coverage and downstream Raw/AR?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Eutd (1/5)

We thank the reviewer for the valuable comments and for recognizing our comprehensive experimental evaluation and continuous exploration during construction. Before responding to each question in detail, we clarify two key mechanisms related to the summary and strengths:

  1. ECC's class-structured behavior model is not a simple hierarchical tree, but a typed multirelational graph with four edge types—inheritance, ownership, reference, and call—that organizes behavioral hierarchy through high-level classes and fine-grained subclasses while capturing cross-module dependencies and call interactions. Moreover, the model is not static: as new tests are assigned via the lowest common ancestor (LCA) rule and classes exceeding the load limit automatically split into subclasses, the model co-evolves with the test suite, improving BehaviorDistill's source-code coverage over Gongent by up to 38.4 percentage points and surpassing the official test suite on MirrorCode at 51.1% (Secs. 2.2.1–2.2.2, 3.2; Tables 1, 5).
  2. PCC uses a dynamic plan to decompose ECC's behavior model and test scripts with expected outputs removed into staged tasks, while continuing to probe the teacher during construction to expand its self-generated test suite and guide implementation and repair from scratch—co-evolving the preliminary specification, dynamic plan, self-generated test suite, and student code. Consequently, combining preliminary ECC specifications with PCC alone raises overall Raw/AR from 71.6%/28.0% to 77.5%/32.0%, and Hard Raw from 43.2% to 56.7% (Secs. 2.3, 3.4; Appendix Tables 4, 6).
AuthorsOpenReview ↗

Response to Reviewer Eutd (2/5)

[W1 & Q1] Design Logic and Running Example for Structural Updates, Class Load, and Test Assignment

We have streamlined the auxiliary mathematical notation and incorporated a running example in Section 2.2 of the revised manuscript. Below, as requested, we use the File I/O module of the DuckDB CLI (Appendix Figure 4) as a running example to intuitively illustrate how these three mechanisms evolve:

  1. Structural Updates: We organize tests in a class-structured behavior model to balance exploration breadth and depth; because few tests are available early on to build the complete hierarchy upfront, the model must dynamically evolve from coarse to fine as new tests arrive (Secs. 1, 2.2; Appendix Figure 4). Software replication requires executable tests to verify behavior, yet unstructured tests obscure functional relationships and coverage gaps. We therefore organize tests within a class-structured behavior model, where high-level classes guide the discovery of uncovered major features for breadth and fine-grained subclasses drive the exploration of boundary cases and feature interactions for depth. Early in exploration, however, only a few tests are available, supporting only a handful of coarse high-level classes rather than all fine-grained subclasses at once. As new tests expose finer behavioral distinctions, the model undergoes continual structural updates—splitting coarse classes into fine-grained subclasses and adding functions and cross-class relations while keeping existing test inputs and outputs unchanged (Supplementary Formal Proofs, Prop. 2.4). For example, after accumulating tests, DuckDB's top-level File I/O class splits by behavioral differences into six subclasses—including CSV exchange, Parquet exchange, and I/O recovery—guiding agents to probe boundary behaviors within each subclass.
  2. Class Load: To trigger these structural updates and prevent the model from degenerating into a coarse, shallow hierarchy, we bound per-class capacity (Sec. 2.2.2 B, D; Appendix Figure 4). Without a per-class capacity limit, the agent would keep accumulating new tests inside a few top-level classes instead of splitting subclasses, leaving the model coarse and shallow without fine-grained subclasses to guide deep exploration. We therefore define a class's load as the sum of its uniquely assigned tests and direct subclasses, bounded by a per-class load limit NcN_c; exceeding NcN_c triggers the structural split above. For instance, before splitting, DuckDB's CSV read/write, Parquet scan, and error-recovery tests all accumulate under File I/O; once its uniquely assigned tests plus direct subclasses exceed NcN_c, splitting is automatically triggered.
  3. Test Assignment (LCA Assignment): Because our class-structured behavior model is a multirelational graph combining a class inheritance tree with function ownership, cross-class reference, and function call edges, a single test can link to multiple classes at once and prevent direct load calculation; we therefore introduce the LCA rule to assign each test to a unique class for class-specific test partitioning and load calculation (Secs. 2.2.1–2.2.2; Appendix Figure 4). Specifically, along the inheritance tree, a test linked to a single class is assigned directly to that class, whereas a test linked to multiple classes is uniquely assigned to their lowest common ancestor (LCA), contributing only to its assigned class's load. In DuckDB, for example, a test exercising only CSV reading/writing is assigned to the subclass CSV exchange and counted toward its load; by contrast, an error-recovery test spanning both CSV exchange and I/O recovery is assigned via LCA to their common parent File I/O and counted solely toward File I/O's load.
  4. Why Each Test Belongs to Exactly One Class: so that each level holds only tests of its own granularity, which breadth–depth exploration relies on (Secs. 2.2.1–2.2.2).
    • Too high floods high-level classes: if a test could belong to every related class, the root would hold all tests, and File I/O would be filled with CSV-quoting and Parquet edge cases, hiding which major file-I/O features are missing.
    • Too low pollutes subclasses: putting a cross-class test into just one of its classes, e.g., the CSV-recovery test into CSV exchange, mixes recovery behavior into a CSV-only class.
    • The LCA is just right: the CSV-recovery test goes to File I/O and CSV-only tests stay in CSV exchange, while ρ\rho keeps every test linked to all related classes.
AuthorsOpenReview ↗

Response to Reviewer Eutd (3/5)

[W2 & Q2] Actual Token/Time Consumption and Performance Comparison under Matched Total Budgets

We fix the cumulative two-stage budget to 200M tokens per task, and in the continuation experiments of Appendix Table 3, include the 6h preliminary stage to align methods at matched total wall-clock budgets of 12h, 18h, and 24h: under an equal 200M total token budget, BehaviorDistill still outperforms the strongest baseline by 4.9 and 4.5 percentage points in Raw and AR; under equal total wall-clock times of 12h–24h, it leads the strongest baseline in AR by 7.5–8.5 percentage points across all checkpoints (Secs. 3.3–3.4; Appendix Tables 3, 7):

  1. Under a fixed 200M total token budget, BehaviorDistill leads across both Raw and AR (Sec. 3.4; Table 1; Appendix Table 7). Appendix Table 7 fixes the cumulative token budget across both exploration and construction to 200M tokens per task with no time limit. Following your suggestion, we additionally measured the per-task mean actual total token usage in the main experiments on ProgramBench (DeepSeek-V4-Flash-0731), as shown below:
MethodEqual 200M Total Tokens Raw / AR (%, App. Table 7)Main Exp. Actual Total TokensMain Exp. Raw / AR (%, Table 1)
Vanilla71.8 / 24.0106.8M69.3 / 18.5
SpecFirst73.3 / 27.5178.6M71.5 / 25.0
Gongent69.7 / 23.5154.2M70.6 / 21.5
BehaviorDistill78.2 / 32.0182.4M77.5 / 32.0
  • Under the equal 200M total token budget, BehaviorDistill achieves 78.2% Raw and 32.0% AR, outperforming the strongest baseline SpecFirst by 4.9 and 4.5 percentage points and Vanilla by 6.4 and 8.0 percentage points.
  • In the main experiments, Vanilla and Gongent terminate prematurely after implementing surface functionality due to the lack of fine-grained test feedback, consuming only 106.8M and 154.2M tokens and stalling at 18.5% and 21.5% AR; BehaviorDistill, driven by PCC's dynamic plan and dual-suite validation to continually repair deep boundary defects, uses 182.4M tokens—comparable to the 178.6M tokens of SpecFirst, which also includes two stages.
  • Even when all baselines are forced to continue up to 200M tokens in Appendix Table 7, their Raw and AR still fall short of BehaviorDistill in the main experiments using only 182.4M tokens: Vanilla and SpecFirst gain little from extra tokens because they lack executable tests to locate deep discrepancies; Gongent relies on a static test suite without PCC's dynamic planning or construction-time test expansion, so forced continuation overfits narrow local tests and breaks uncovered behaviors, reducing its Raw from 70.6% to 69.7%; by contrast, BehaviorDistill leverages PCC to continually expand tests and perform staged repairs during construction, turning additional tokens into a higher Raw of 78.2%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.