- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the BehaviorDistill framework. It employs "Evolving Class-Structured Coverage" (ECC) to construct structured behavioral specifications that evolve continuously throughout the testing process. Subsequently, it utilizes "Plan-Guided Coverage-Expanding Construction" (PCC) to organize the implementation of tests and the repair of code based on these specifications, thereby enabling the behavioral model, the test suite, and the software implementation to co-evolve throughout the exploration process. The problem holds practical significance; the proposed method bridges "exploration" and "construction," and its effectiveness has been verified through experiments.
Critical strengths
- A black-box software specification method is proposed that integrates executable tests with structured behavioral models. This approach allows specifications to not only guide development but also facilitate verification, thereby enabling the automated testing and repair of software code.
- BehaviorDistill introduces two processes: ECC (Explore and Construct Structured Behavioral Specifications) and PCC (Implement based on these specifications, while continuing to uncover omissions during implementation), to automatically explore the behavior of a black-box program and build, from scratch, a "student" program that exhibits similar behavior.
Critical weaknesses
The problem definition and formulation require further refinement. For instance, what are the specific scenarios or types of software for which the proposed method is intended? Additionally, what is the rationale behind the class structure/behavioral models and behavioral specifications? Could you provide a more detailed explanation of your approach?
Questions for authors
- How is the construction of behavioral models and ECC coverage managed to ensure that it does not negatively impact the performance of students' subsequent programs? Is there a performance-related correlation between the two?
- The paper mentions "scenarios where software source code is not accessed"。In such cases, does the source code have any bearing on the exploration of ECC or the construction of PCC?
7 comments
Response to Reviewer HokB (2/4)
- The design motivation for behavioral specifications and the class-structured behavior model was systematically presented in the Abstract, Section 1 (Paragraphs 2–3), and Section 2.2.1 (Paragraphs 1–3) of the original submission:
- Software documentation is often coarse-grained and leaves detailed feature interactions and boundary cases unspecified, requiring agents to actively probe the black-box teacher and abstract its behavior into a specification to guide replication. Prior approaches such as SpecFirst rely on natural-language specifications, which cannot be executed to verify behavioral consistency or pinpoint discrepancies; behavioral specifications must therefore include executable tests (Abstract; Sec. 1).
- However, while unstructured collections of tests verify individual observed behaviors, they obscure behavioral subspaces and their relations within a vast behavior space; a behavior model is therefore needed to organize tests structurally, locate coverage gaps, and balance exploration breadth and depth under a finite budget (Abstract; Secs. 1, 2.2.1).
- Conventional behavior models such as state machines and input classification trees typically focus on a single behavioral dimension, rely on manual abstraction, and lack a unified representation for direct composition and refinement. We therefore represent the class-structured behavior model as a code skeleton containing only class and function declarations: classes represent behavior subspaces, functions represent operations, and inheritance, ownership, reference, and call edges form a typed multirelational graph that losslessly unifies and composes state machines, input classification trees, and dependency graphs; each test is uniquely assigned to one class and linked to relevant classes or functions, forming the behavioral specification together with the model (Sec. 2.2.1; Supplementary Formal Proofs, Theorem 2.2, Prop. 2.3).
- The algorithmic workflow and co-evolving artifacts of ECC and PCC were detailed in Figure 1, Section 1 (Paragraphs 4–6), and Sections 2.2–2.3 of the original submission:
- ECC is the coverage-driven exploration stage, in which the behavior model and the test suite co-evolve while probing the teacher (Sec. 2.2.2). Exploration Agents first query the teacher to generate initial tests and synthesize an initial class structure, then use the class structure as a coverage map to explore across levels in parallel: high-level classes discover uncovered major features, while fine-grained subclasses probe input boundaries, state changes, and feature interactions. New tests are assigned to classes via the lowest common ancestor (LCA) rule and trigger subclass splitting when a class exceeds the load limit, without relying on handcrafted test dimensions or combination rules as in Gongent (Secs. 1, 2.2.2).
- PCC is the student construction stage, during which the ECC model and test suite remain fixed, while the dynamic plan, the Builder's self-generated test suite, and the student code co-evolve (Sec. 2.3). Guided by the behavior model and test scripts, the Builder uses a dynamic plan to organize test constraints into staged student implementation and repair tasks, then iteratively implements or repairs code according to the plan, probes the teacher on uncovered areas to expand its self-generated test suite, executes both test suites, and revises the plan from test feedback until student construction is complete (Sec. 2.3). Using black-box exploration alone, our method reaches 51.1% test coverage on MirrorCode—surpassing the human-revised official suite (47.1%)—and improves Raw and AR over Vanilla by up to 15.8 and 18.2 percentage points (Table 1).
Response to Reviewer HokB (3/4)
[Q1] How Are Behavior Model Construction and ECC Coverage Managed to Avoid Negative Effects, and Do They Correlate with Student Performance?
- Sections 2.2.2, 2.3, and 3.4 and Appendix Table 4 of the original submission explicitly detailed three mechanisms that prevent specifications from negatively affecting student construction:
- Every test in ECC must be executed on the real teacher with its actual output captured before entering the suite, and the class-structured behavior model serves solely as a behavioral coverage map without constraining the internal architecture of the student implementation (Secs. 2.2.2, 2.3; Supplement 03).
- PCC separates the fixed ECC suite into input-only test scripts and hidden expected outputs stored in a validation environment; the Builder can inspect only the input scripts and receives pass/fail feedback through test execution, preventing hardcoded outputs (Sec. 2.3).
- Handing a large suite of fine-grained tests all at once to an unplanned builder can indeed overwhelm it—for instance, in Table 4,
ECC + Vanillasees Hard Raw drop fromVanilla's 45.3% to 41.8%. PCC resolves this by using a dynamic plan to decompose large test suites into staged implementation and repair tasks, lifting Hard Raw to 56.7% and overall performance to 77.5% Raw and 32.0% AR (Secs. 2.3, 3.4; Appendix Table 4).
- Section 3.2 (Table 1) and Section 3.4 (Appendix Tables 4 and 5) of the original submission, together with our supplementary experiments, empirically demonstrate that ECC coverage correlates positively with student replication performance and translates fully into gains when paired with PCC:
- Across all four experimental settings in Table 1 of the original submission, BehaviorDistill achieves substantially higher test coverage than Gongent alongside higher Raw and AR across the board (Sec. 3.2; Table 1). Moreover, when the three exploration variants in Appendix Table 5—
Flat,Tree-only, andFull ECC(whose exploration coverage increases from 58.5% to 63.9% to 69.8%)—are paired with the identical PCC stage, downstream Raw/AR rises monotonically from 71.7%/27.5% to 73.8%/29.0% to 77.5%/32.0% (see the supplementary experiment in our response to Reviewer U7xW[W1 & Q1]), and across the 200 tasks of ProgramBench (DeepSeek-V4-Flash-0731), ECC test coverage exhibits a Spearman rank correlation of 0.64 with hidden-test pass rate. - At the same time, high coverage alone is not sufficient: in Table 4,
ECC + Vanillaachieves higher coverage (69.8%) thanPCC only(62.6%) but lower Raw and AR, demonstrating that high-coverage specifications must be paired with PCC's staged planning mechanism to translate effectively into student replication performance (Sec. 3.4; Appendix Table 4).
- Across all four experimental settings in Table 1 of the original submission, BehaviorDistill achieves substantially higher test coverage than Gongent alongside higher Raw and AR across the board (Sec. 3.2; Table 1). Moreover, when the three exploration variants in Appendix Table 5—
Response to Reviewer HokB (1/4)
We thank the reviewer for the valuable comments and for recognizing the practical significance and method design of our work. The target application scenario, motivation for behavioral specifications and the class-structured behavior model, synergy between the two stages, and strictly source-free black-box setting were all explicitly stated and empirically validated in the Abstract, Sections 1–3, and Appendices C.1–C.4 of the original submission. Below, we clarify each point with explicit references to the submitted manuscript:
[Paper Summary, Critical Strengths & W1] Target Scenario and Software Types, Motivation for Behavioral Specifications and Class-Structured Behavior Models, and Method Overview