- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ShapingBench studies how much mature software behavior coding agents reproduce when given public implementation tasks without access to individual evaluation contracts. The benchmark covers nine software domains and 45 reference implementations, producing 14,537 executable contracts. Contracts are validated on their source implementations and organized by how many of five references reproduce them, separating a Shared Core from a Variable Surface. The authors evaluate four LLMs in ten model–framework configurations across three agent frameworks, yielding 90 development trajectories. They report Shared Core coverage of 62.1–88.2% and lower coverage at lower recurrence levels across all ten configurations. Continued development, including sessions with broad refinement hints, leaves substantial residual gaps. The paper also presents contract-construction examples, per-domain analyses, and an oracle audit that detects 96.8% of valid mutants affecting contract-required observations. Its contribution is an executable evaluation resource and an empirical characterization of behavioral coverage across implementation recurrence.
Critical strengths
S1: Organizing executable software behaviors by cross-implementation recurrence provides a useful diagnostic beyond an aggregate pass rate. The distinction between Shared Core and Variable Surface is clearly defined, and Table 2 shows the same direction of coverage gradient across all ten evaluated configurations. The remaining Shared Core gap is relevant to assessing the reliability of software generated by coding agents. I also appreciate the explicit qualification that recurrence measures prevalence among the selected implementations rather than importance, and that full Variable Surface coverage need not be a natural optimum.
S2: The benchmark spans nine domains and 45 reference projects, while the experiments include four models, three agent frameworks, and 90 development trajectories. Source replay, expectation-perturbation checks, cross-implementation replay, and the additional behavioral-mutant audit provide complementary evidence for the evaluation machinery. Appendix G transparently reports surviving mutants and audit coverage limits. The domain-level analyses and worked construction example also help readers assess the reported results. These features support the value of the benchmark as a reusable evaluation resource.
Critical weaknesses
W1: The public URL task in Appendix B.3 explicitly requires resolving relative references against a base URL. The worked Shared Core contract in Appendix F tests this same behavior with a particular relative input and base. Although that exact test instance is hidden, its behavioral requirement is already stated. The observed Shared Core gap may therefore combine failures to implement explicit requirements with failures to infer additional behavior. This distinction matters for the paper’s central interpretation of evaluation beyond explicit requirements. A modest, stratified audit of residual Shared Core failures, distinguishing behaviors stated or directly entailed by the public task, behaviors subsequently covered by hints, and behaviors absent from both, would help establish how much of the gap supports the intended interpretation. I am not asking for the exact hidden tests to be disclosed to the agents.
W2: Table 1 includes both Requests and urllib3, although the transport adapter in the reported Requests version uses urllib3 (https://raw.githubusercontent.com/psf/requests/v2.32.5/src/requests/adapters.py). It also includes PyYAML and ruamel.yaml, whose documentation identifies a PyYAML lineage (https://yaml.dev/doc/ruamel.yaml/overview/). These relationships do not invalidate the observation that all five selected projects pass a contract, but they weaken its interpretation as five independent sources of behavioral evidence and may affect recurrence strata. The paper should clarify what independence means here and assess sensitivity to the known relationships. A targeted analysis using the existing support matrix, omitting one related reference at a time in the affected domains and recomputing the corresponding recurrence strata, would be sufficient to address this concern without rerunning the entire agent study.
Questions for authors
Q1: How do you distinguish an unstated behavioral requirement from a hidden test instance of an explicitly required behavior, such as the relative-resolution example in Appendices B.3 and F? Could you provide a stratified audit of remaining Shared Core failures that separates requirements already stated or entailed by the public task, those covered by later hints, and those absent from both?
Q2: How is reference independence defined given the Requests–urllib3 dependency and PyYAML–ruamel.yaml lineage? Using the existing results, does the recurrence–coverage pattern remain qualitatively similar when one member of each known related pair is omitted in the affected domain and recurrence is recomputed?
2 comments
Post-Rebuttal Assessment
Thank you for the clarifications and additional analyses. The failure audit addresses my concern about distinguishing hidden test instances from unstated behavioral requirements and supports a more qualified interpretation of the Shared Core gap. The leave-one-reference-out results also substantially alleviate my concern about the influence of related references on the recurrence–coverage pattern. I appreciate the explicit acknowledgement of the sampling limitations and the distinction between distinct projects and independent implementation lineages. Please incorporate these analyses and qualifications into the revised manuscript. These responses strengthen my positive assessment of the benchmark, and I maintain my original recommendation of weak acceptance.
Rebuttal (Part 1/1)
We thank the reviewer for the thorough assessment and constructive feedback on our work.
Hidden test instances and unstated requirements are distinct. Relative URL resolution is explicitly required in Appendix B.3; Appendix F illustrates contract construction and validation, not an unstated requirement.
To examine this distinction, we audited 198 failed Shared Core evaluation instances corresponding to 188 distinct scoring IDs, sampling 22 instances from each of the nine domains across behavioral families. We classified the tested behaviors against the initial tasks and subsequent refinement hints.
The results show that Shared Core failures include both explicitly required and unstated behaviors. These proportions describe the domain-balanced sample, not the full failure population. We will clarify this distinction rather than attributing all failures to implicit requirements.
We clarify that the references are distinct projects, but they do not necessarily provide independent implementation lineages. We will discuss this distinction and document the known relationships in the revised manuscript.
We conducted a leave-one-reference-out sensitivity analysis using the original support matrix and final agent evaluations.
Values are mean coverage percentages across ten configurations per condition. All four mean recurrence-coverage trends remain strictly monotonic. At the individual-configuration level, strict monotonicity holds in 35/40 cases, while all 40 retain higher coverage at 4/4 than at 1/4.
The original 5/5 Shared Core is contained in each recomputed 4/4 core. Behaviors supported only by the omitted reference (0/4) remain in full-surface coverage but are excluded from the recurrence gradient.
These results indicate that the main empirical pattern is robust to excluding either related reference, although complete independence cannot be established. A complementary sixth-reference validation in Markdown also supports Shared Core stability in that domain (as detailed in our response to Reviewer
rbHH, Q6).