acceptodds
Under review as a conference paper at ICLR 2027

ShapingBench: Can Coding Agents Reproduce Mature Software Behavior Beyond Explicit Requirements?

Abstract

Coding agents are becoming increasingly capable of implementing software from high-level requirements, often expressed in natural language. However, existing benchmarks mainly evaluate behavior that is explicitly specified in the underlying task. We instead study how much of the behavior accumulated in mature software an agent can realize when those expectations are never individually stated. We introduce ShapingBench, a benchmark spanning 9 software domains, 45 mature open-source implementations, and 14,537 executable behavioral contracts. Behaviors are recovered from five independently developed implementations per domain and organized by how widely they recur across them. Behaviors shared by all five form the Shared Core, while the remaining behaviors form the Variable Surface. More than half of the Variable Surface appears in at least two reference implementations. We evaluate 4 LLMs across 3 agent frameworks, covering 10 model and framework combinations and 90 development trajectories. Behavioral coverage decreases consistently with cross-implementation recurrence, ranging from 62.1%-88.2% for behaviors shared by all five references to 12.4%-21.3% for behaviors observed in only one. Additional development improves coverage but yields diminishing gains, while many remaining failures involve behaviors repeatedly found across independent implementations. These results reveal a persistent gap between implementing explicit requirements and realizing the hidden behavioral expectations exhibited by mature software.

Primary area:
datasets and benchmarks
Keywords:
code generation, coding agents, software evolution, benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5YyoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ShapingBench studies how much mature software behavior coding agents reproduce when given public implementation tasks without access to individual evaluation contracts. The benchmark covers nine software domains and 45 reference implementations, producing 14,537 executable contracts. Contracts are validated on their source implementations and organized by how many of five references reproduce them, separating a Shared Core from a Variable Surface. The authors evaluate four LLMs in ten model–framework configurations across three agent frameworks, yielding 90 development trajectories. They report Shared Core coverage of 62.1–88.2% and lower coverage at lower recurrence levels across all ten configurations. Continued development, including sessions with broad refinement hints, leaves substantial residual gaps. The paper also presents contract-construction examples, per-domain analyses, and an oracle audit that detects 96.8% of valid mutants affecting contract-required observations. Its contribution is an executable evaluation resource and an empirical characterization of behavioral coverage across implementation recurrence.

Critical strengths

S1: Organizing executable software behaviors by cross-implementation recurrence provides a useful diagnostic beyond an aggregate pass rate. The distinction between Shared Core and Variable Surface is clearly defined, and Table 2 shows the same direction of coverage gradient across all ten evaluated configurations. The remaining Shared Core gap is relevant to assessing the reliability of software generated by coding agents. I also appreciate the explicit qualification that recurrence measures prevalence among the selected implementations rather than importance, and that full Variable Surface coverage need not be a natural optimum.

S2: The benchmark spans nine domains and 45 reference projects, while the experiments include four models, three agent frameworks, and 90 development trajectories. Source replay, expectation-perturbation checks, cross-implementation replay, and the additional behavioral-mutant audit provide complementary evidence for the evaluation machinery. Appendix G transparently reports surviving mutants and audit coverage limits. The domain-level analyses and worked construction example also help readers assess the reported results. These features support the value of the benchmark as a reusable evaluation resource.

Critical weaknesses

W1: The public URL task in Appendix B.3 explicitly requires resolving relative references against a base URL. The worked Shared Core contract in Appendix F tests this same behavior with a particular relative input and base. Although that exact test instance is hidden, its behavioral requirement is already stated. The observed Shared Core gap may therefore combine failures to implement explicit requirements with failures to infer additional behavior. This distinction matters for the paper’s central interpretation of evaluation beyond explicit requirements. A modest, stratified audit of residual Shared Core failures, distinguishing behaviors stated or directly entailed by the public task, behaviors subsequently covered by hints, and behaviors absent from both, would help establish how much of the gap supports the intended interpretation. I am not asking for the exact hidden tests to be disclosed to the agents.

W2: Table 1 includes both Requests and urllib3, although the transport adapter in the reported Requests version uses urllib3 (https://raw.githubusercontent.com/psf/requests/v2.32.5/src/requests/adapters.py). It also includes PyYAML and ruamel.yaml, whose documentation identifies a PyYAML lineage (https://yaml.dev/doc/ruamel.yaml/overview/). These relationships do not invalidate the observation that all five selected projects pass a contract, but they weaken its interpretation as five independent sources of behavioral evidence and may affect recurrence strata. The paper should clarify what independence means here and assess sensitivity to the known relationships. A targeted analysis using the existing support matrix, omitting one related reference at a time in the affected domains and recomputing the corresponding recurrence strata, would be sufficient to address this concern without rerunning the entire agent study.

Questions for authors

Q1: How do you distinguish an unstated behavioral requirement from a hidden test instance of an explicitly required behavior, such as the relative-resolution example in Appendices B.3 and F? Could you provide a stratified audit of remaining Shared Core failures that separates requirements already stated or entailed by the public task, those covered by later hints, and those absent from both?

Q2: How is reference independence defined given the Requests–urllib3 dependency and PyYAML–ruamel.yaml lineage? Using the existing results, does the recurrence–coverage pattern remain qualitatively similar when one member of each known related pair is omitted in the affected domain and recurrence is recomputed?

2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/1)

We thank the reviewer for the thorough assessment and constructive feedback on our work.

[W1, Q1] Explicit Requirements and Residual Shared Core Failures

Hidden test instances and unstated requirements are distinct. Relative URL resolution is explicitly required in Appendix B.3; Appendix F illustrates contract construction and validation, not an unstated requirement.

To examine this distinction, we audited 198 failed Shared Core evaluation instances corresponding to 188 distinct scoring IDs, sampling 22 instances from each of the nine domains across behavioral families. We classified the tested behaviors against the initial tasks and subsequent refinement hints.

ClassificationInstancesShare
Explicitly stated in the initial task2914.6%
Directly entailed by the initial task3417.2%
Covered only by later hints3618.2%
Unstated in both5829.3%
Ambiguous4120.7%
Total198100.0%

The results show that Shared Core failures include both explicitly required and unstated behaviors. These proportions describe the domain-balanced sample, not the full failure population. We will clarify this distinction rather than attributing all failures to implicit requirements.

[W2, Q2] Reference Independence and Sensitivity Analysis

We clarify that the references are distinct projects, but they do not necessarily provide independent implementation lineages. We will discuss this distinction and document the known relationships in the revised manuscript.

We conducted a leave-one-reference-out sensitivity analysis using the original support matrix and final agent evaluations.

Excluded Reference1/42/43/44/4
HTTP: urllib37.0%36.4%46.6%71.6%
HTTP: Requests4.8%34.3%47.3%71.9%
YAML: PyYAML14.3%46.4%62.2%86.0%
YAML: ruamel.yaml24.3%42.4%62.6%87.5%

Values are mean coverage percentages across ten configurations per condition. All four mean recurrence-coverage trends remain strictly monotonic. At the individual-configuration level, strict monotonicity holds in 35/40 cases, while all 40 retain higher coverage at 4/4 than at 1/4.

The original 5/5 Shared Core is contained in each recomputed 4/4 core. Behaviors supported only by the omitted reference (0/4) remain in full-surface coverage but are excluded from the recurrence gradient.

These results indicate that the main empirical pattern is robust to excluding either related reference, although complete independence cannot be established. A complementary sixth-reference validation in Markdown also supports Shared Core stability in that domain (as detailed in our response to Reviewer rbHH, Q6).

Reviewer_5YyoOpenReview ↗

Post-Rebuttal Assessment

Thank you for the clarifications and additional analyses. The failure audit addresses my concern about distinguishing hidden test instances from unstated behavioral requirements and supports a more qualified interpretation of the Shared Core gap. The leave-one-reference-out results also substantially alleviate my concern about the influence of related references on the recurrence–coverage pattern. I appreciate the explicit acknowledgement of the sampling limitations and the distinction between distinct projects and independent implementation lineages. Please incorporate these analyses and qualifications into the revised manuscript. These responses strengthen my positive assessment of the benchmark, and I maintain my original recommendation of weak acceptance.

Reviewer_c83ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ShapingBench, a benchmark designed to evaluate the extent to which coding agents can reproduce behaviors exhibited by mature software that are not explicitly specified, given only high-level task descriptions and interface requirements. The authors recover 14,537 executable behavioral contracts from 45 mature open-source implementations across 9 software domains and categorize them into Shared Core and Variable Surface based on cross-implementation recurrence. The paper validates these contracts through source implementation replay, oracle sanity checks, and behavioral-mutant validity audits, and evaluates 10 agent–LLM configurations. The results show that no evaluated configuration achieves complete Shared Core coverage, and behavioral coverage consistently decreases as cross-implementation recurrence declines. Additional development iterations and broad refinement guidance also fail to fully close this gap.

Critical strengths
  1. Novel evaluation perspective. The paper evaluates coding agents' ability to reproduce mature software behaviors beyond explicit requirements. Its cross-implementation recurrence framework provides a more nuanced analysis of behavioral coverage.
  2. Rigorous benchmark construction and validation. The benchmark covers multiple software domains with extensive executable behavioral contracts. Source replay, oracle sanity checks, and behavioral-mutant audits strengthen its reliability.
Critical weaknesses
  1. Behaviors found in mature software are not necessarily requirements for the current task. The benchmark relies on mature software behavior to assess the completeness of new implementations. However, agreement across all five references does not establish that a behavior is required by the current task; it may reflect shared conventions or legacy compatibility choices. Behaviors unique to one reference may simply be library-specific extensions. The paper should better justify why these behaviors constitute appropriate evaluation targets.
  2. The evaluation conflates prior knowledge, implicit requirement inference, and implementation ability. The contracts are derived from mature open-source projects whose code, documentation, tests, and associated standards may have appeared in model training data. Successful reproduction may therefore reflect learned domain knowledge or memorized implementations rather than inference of implicit requirements from the current task context. Conversely, contract failures may stem from implementation errors rather than failure to identify the requirements. The paper should clarify which capabilities the benchmark measures and how these explanations can be distinguished.
  3. Insufficient clarity in task and behavioral contract construction. The paper includes only nine development tasks but provides limited details on their design . Moreover, the example in the appendix relies on keyword matching and predefined tests, while the overall extraction rules remain unclear.
  4. The effects of additional development effort and requirement hints remain confounded. Across iterations, agents spend more time developing the software and receive hints that clarify the expected behavior. Although the authors acknowledge that these comparisons are descriptive, without comparing agents with and without hints under the same development budget, it is difficult to tell whether the gains come from more coding and debugging or clearer requirements. It is also unclear whether the hints help agents identify missing requirements or fix implementation errors.
Questions for authors
  1. How do the authors distinguish required behaviors, optional extensions, and legitimate implementation differences based on the task description or applicable standards? What evidence, beyond cross-implementation recurrence, supports these contracts as appropriate evaluation targets for new software?
  2. How can the evaluation distinguish reliance on pretrained domain knowledge or memorized implementations from inference of implicit requirements from the current task context? For failed contracts, can the authors separate requirement identification failures from implementation errors to clarify which capabilities the benchmark measures?
  3. Could the authors clarify how the nine tasks were designed and how public requirements were separated from hidden contracts? Were the contracts extracted using predefined rules, LLMs, or human annotation?
  4. Could the authors compare continued development with and without refinement hints under the same development budget to isolate the contributions of additional effort and requirement information?
3 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/1)

We thank the reviewer for the detailed and constructive feedback.

[W1, Q1] Behavioral Expectations vs. Requirements

We clarify that cross-implementation recurrence does not establish normative requirements. As defined in Section 3.1, ShapingBench uses recurrence to measure empirical consistency rather than absolute behavioral importance. Full-surface coverage is designed as a descriptive measure rather than a rigid requirement to implement every observed behavior. Consequently, the Shared Core provides a reproducible, cross-implementation reference for assessing behavioral coverage under underspecified development tasks, without assuming that every shared behavior is mandatory.

To examine the stability of this empirical agreement, we added a sixth native Markdown implementation (Marko 2.2.5). As shown in our response to Reviewer rbHH Q6, 1,091 out of 1,114 (97.94%) of the original Shared Core contracts were retained. This supports the stability of the observed agreement in the selected domain.

[W2, Q2] Behavioral Coverage and Failure Attribution

ShapingBench evaluates end-to-end behavioral reproduction: how much mature-software behavior an agent realizes from a public task without receiving the concrete behavioral contracts (Section 3.1 and 4). Prior knowledge, requirement inference, and implementation ability can all contribute; contract outcomes alone do not disentangle these mechanisms.

Our core findings concern the resulting software: no evaluated configuration fully reproduces the Shared Core, and coverage consistently decreases as cross-implementation recurrence falls across all ten configurations (Table 2). These findings remain valid without determining how each behavior was acquired or why each contract failed. ShapingBench thus establishes a gap between implementing a high-level task and reproducing mature software behavior, without attributing that gap to a single underlying mechanism.

[W3, Q3] Task and Contract Construction

The nine domains were selected from twelve explored domains. Reference selection followed qualitative criteria including comparable functionality, release histories, replayability, and ecosystem diversity (Section 3.2; Appendix A).

Contract construction combined rule-based extraction from release histories, transformation of upstream tests and fixtures, and curated behavioral scenarios. In the Appendix F example, release-note keywords trigger predefined behavioral cases. Expected observations came from existing fixtures, explicit assertions, or pinned source implementations. Contracts were validated through source replay, oracle sanity checks, and cross-implementation replay (Section 3.1; Appendix F).

Coding agents assisted in developing extraction tools, but the inspected runtime extractors did not invoke LLM APIs for per-contract generation. We do not claim exhaustive human annotation. Public tasks specified functionality and interfaces, while individual evaluation contracts were maintained separately (Appendix B).

[W4, Q4] Development Effort and Refinement Hints

We supplemented the descriptive comparison in §4.2 with a targeted pilot on Date/Time and HTML Sanitization. Starting from identical third-session implementations, we ran one additional Codex/GPT development session per condition, with or without the original refinement hints. The table reports Shared Core coverage:

DomainNo HintWith HintDelta Core (pp)
Date/Time30.8%32.8%+2.00
HTML Sanitization91.5%91.5%0.00

The pilot provides a comparison of hint availability under matched budget limits. Actual token and tool usage differed, and each condition was executed once per domain. We therefore treat these findings as exploratory rather than attributing improvements exclusively to refinement hints.

AuthorsOpenReview ↗

Looking forward to hearing from you

Dear Reviewer,

Thank you again for your time and constructive feedback. As the rebuttal deadline approaches, we would appreciate it if you could let us know whether our responses have addressed your concerns, or if you have any further questions. We would be happy to provide further clarification.

Thanks,

Authors

Thank you for the detailed rebuttal and additional experiments, which address some of my concerns. However, I maintain my rating of 2 (Weak Reject).

The authors clarify that the benchmark evaluates end-to-end behavioral reproduction of mature software, but its significance remains unclear to me. Since commonly observed behaviors are not necessarily required or desirable, what does higher coverage actually tell us about an agent's capabilities?

Additionally, the benchmark includes only nine development tasks, raising concerns about task diversity and generalizability. Overall, I believe these fundamental concerns remain insufficiently addressed.

Reviewer_rbHHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Most benchmarks test agents on implementing very clearly defined specifications. Agents have gotten really good at following such specs, and thus these benchmarks are pretty much all saturating now. This paper therefore thinks about ways to come up with "implicit expectations" that users might have that do not need to be explicitly spelled out. For this they look at cases where the same type of application has several referene implementations and they isolate shared behavior across all. This shared behavior then should be expected from the agent even if it isn't told

Critical strengths

Models are already extremely good at implementing given specs. The remaining bottlenecks are e.g., in how models interpret more ambiguous/vague instructions. This is exactly what this paper targets. The idea is novel and creative and leads to a clear (if possibly somewhat domain-limited) research setup.

Critical weaknesses

tl;dr: The benchmark is very small and focuses on a very particular set of domains, and because of its construction it is unlikely that it can be easily broadened to more domains.

  • Very small sample of only 9 tasks
  • Seems to be very hard to standardize across broader domains of software (because it's unusual that there's so many parallel implementation of the exact same "product")
  • For the variable surface, why would we at all expect agents to reproduce such behavior? In fact, splitting this up somewhat by the kind of behavior, depending if it's more "feature"-like vs more "necessary decision" like, my expectation would be
    • feature-like: Agent should only implement if it's in shared core
    • necessary decision: Agent should make the same decision as the majority, i.e., should only implement variable surface of 3/5, 4/5, 5/5, and NOT 1/5, 2/5
    • The paper however seems to have the standpoint that generally "the more the better"
Questions for authors

Questions about experiments

  • Reward hacking: Does the agent have access to internet during the run? Are there any other possible reward hacks (e.g., git history, other information that aren't part of task description)?

Questions about interpretation

  • Any way to control contamination/memorization in this benchmark? Could it be that the score correlates with how much existing implementations etc. have been memorized? Or from a different angle: It seems like most of the examples are around complete specs (e.g., YAML parsing, etc.). Wouldn't it be fair to give the agent access to the YAML reference so it knows exactly how yaml is defined? Could possible do some ablations here around these questions of memorization.
  • Other main question: Is this expression of preferences, e.g., some models focusing on the minimal core of what the users requests rather than others implementing bells and whistles, or is this an expression of capabilities.
    • E.g., let's say every Yaml parser includes a JSON export. I'd consider that not a necessary feature of a YAML parser and I'm not sure if I'd want a LM to create the export without me asking. However probably almost all implementations have just that.
    • Some experiments that would point to benchmark scores being tied with clearly improving capabilities: Increase of performance with increase of reasoning level etc.
    • Is there some qualitative breakdown on what the kind of things are that models are missing? Can we quantify how many of these behaviors

Questions on construction & composition

  • How much does the selection of shared surface change if I add more reference implementations? E..g, if for a few o fthe 9 tasks, we had a 6th reference implementation, we could check which % of the 5-shared core would still apply. Or more broadly, do we have a rough estimate of how close a hypothetical "real" shared core is vs the one that we empirically construct?
4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for the constructive feedback.

[W1, W2] Benchmark Size and Applicability

ShapingBench spans 9 mature software domains, 45 open-source implementations, and 14,537 executable behavioral contracts. This constitutes a high-density behavioral verification space. We choose these domains because infrastructure and core software naturally exhibit ecosystem plurality (multiple mature, independent implementations), enabling the identification of recurring behaviors through cross-implementation comparison. This cross-ecosystem differential approach provides a standardized and rigorous framework for evaluating software behavior.

To explore that this methodology is not restricted to the initial instantiation, we conducted a preliminary survey showing its candidate implementations in additional complex software domains. The table below shows six representative domains, each featuring five candidate open-source implementations spanning at least five distinct programming languages. This indicates that the ShapingBench methodology is scalable and readily extensible to a broader spectrum of software engineering domains.

Candidate domainOpen-source implementationsRepresentative maturity evidence
JSON Schema ValidationAjv, python-jsonschema, jsonschema (Go), jsonschema-rs, JsonSchema.Netpython-jsonschema: v4.26.0 (2026)
DNS Message Processingdnspython, miekg/dns, dnsjava, Hickory DNS, dns-packetdnspython: 2003-2026
PNG Encoding / Decodinglibpng, PNGJ, pngjs, png (Rust), ChunkyPNGlibpng: 31+ years of development
ZIP Archive Processinglibzip, Commons Compress, rubyzip, zip-rs, ADM-ZIPrubyzip: 56 published versions
GraphQL Query Executiongraphql-js, graphql-java, graphql-go, graphql-ruby, async-graphqlgraphql-js: v17.0.2 (2026)
X.509 Certificate ValidationOpenSSL, Bouncy Castle, crypto/x509 (Go), rustls-webpki, node-forgeOpenSSL: 1998-2026

Broader validation remains future work, as establishing comparable behavioral interfaces and validating reference independence across diverse ecosystems requires substantial engineering effort.

[W3, Q3] Variable Surface and Preference vs. Capability

We clarify that not all Variable Surface behaviors are intended to be treated as required functionality. As established in Section 3.1, cross-implementation recurrence measures empirical recurrence rather than behavioral importance or necessity. The 3/5-4/5 strata capture majority-supported behaviors, while the 1/5-2/5 strata characterize less widely shared behaviors. We report all strata to examine how behavioral coverage varies with recurrence, rather than assuming that higher full-surface coverage is always preferable.

Regarding the distinction between feature omission and implementation limitations, ShapingBench measures the observable behavioral coverage of the resulting software, without attributing individual failures to agent preferences or implementation capabilities.

[Q1] Internet Access and External Information

Network access was permitted during development (Section 4; Appendix C). An audit of the preserved traces across all 90 primary trajectories identified evidence of retrieved public information or existing-library results in 39 trajectories. Five final implementations delegated domain functionality to existing library backends, and two used structural parser components.

Public-library reuse complicates attribution of observed coverage to newly implemented behavior. To assess whether the main empirical pattern depends on these identified reuse cases, we excluded all evaluation outcomes from these seven final implementations and recomputed coverage using the existing results.

MetricOriginal (90)Excluding 7 (83)
Shared Core75.13%74.32%
Variable Surface37.11%36.60%
Full Surface48.95%48.22%

The recurrence-coverage trend remained nondecreasing across all ten configurations. This is a case-exclusion sensitivity analysis, not a rerun with dependencies removed. Although exclusion changes the evaluated cohort, these results suggest that the main empirical pattern does not depend on the identified library-reuse cases. ShapingBench evaluates end-to-end behavioral reproduction under these development conditions.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

[Q2] Contamination and Reference Specifications

To test whether providing reference specifications affects performance, we conducted a paired YAML ablation using Codex CLI and GPT-5.6-Sol (high), providing the complete YAML 1.2.2 specification directly in the treatment's initial context.

MetricControlWith Specification
Shared Core passes (n = 1,881)1,8801,880
Shared Core coverage99.95%99.95%
Full-surface coverage72.18%72.15%

Although aggregate coverage was nearly identical, 289 contract outcomes differed: 144 treatment-only and 145 control-only passes.

This empirical ablation shows no aggregate improvement from specification provision under the tested conditions. Both implementations used PyYAML as a backend, and near-ceiling Shared Core performance limits interpretation. The result does not establish the absence of memorization or a general effect of specification access.

[Q4] Reasoning Effort

We extended the same-model comparison to four domains using GPT-5.6-Sol with medium and high reasoning effort under matched Codex settings. We report the validated final implementations for each domain and effort condition in the table below, which details the Shared Core coverage.

DomainMediumHighDelta (pp)
Cron94.92%97.18%+2.26
HTML Sanitization87.23%93.62%+6.38
URL/IRI90.91%90.91%0.00
JWT100.00%100.00%0.00
4-domain macro93.26%95.43%+2.16

High reasoning effort yielded higher Shared Core coverage on Cron and HTML Sanitization, while URL/IRI and JWT remained unchanged. Full-surface coverage also increased by 1.30 percentage points on average. These results provide exploratory evidence that behavioral coverage can improve with increased reasoning effort.

[Q5] Behavioral Categories of Residual Failures

We analyzed 145,370 final evaluation outcomes (14,537 contracts across ten configurations) using the nine behavioral categories defined in Section 3.2 (Figure 2). The table shows selected categories; each percentage is the category's share of failure occurrences within that surface, not its within-category failure rate.

Behavioral CategoryShare of Shared Core FailuresShare of Variable Surface Failures
Standards / interoperability / compatibility25.83%14.79%
Primary functional semantics19.17%10.88%
Input validation / error behavior16.35%8.56%
Security / safety3.68%14.40%
Representation / formatting / round-trip15.07%13.92%

Standards, primary functionality, and input validation account for approximately 61% of Shared Core failures. Variable Surface failures are distributed across multiple categories, including standards, security, and representation.

A qualitative examination of 45 selected cases further identified missing behavior, representation mismatches, and execution timeouts. Residual failures are therefore not confined to extension-category behavior; they also occur in standards-related, primary-functional, and validation categories.

[Q6] Shared Core Stability with a Sixth Reference

We conducted a targeted sixth-reference validation on Markdown, selecting the domain before observing replay outcomes. We added Marko 2.2.5, a distinct native Python implementation, and replayed all 2,159 frozen contracts without modifying the original benchmark.

OutcomeOriginal 5/5 Shared Core (n = 1,114)
PASS1,091
Not retained23
Confirmed retention97.94%

Regrouping existing agent outcomes under the confirmed 6/6 Core of 1,091 contracts changed mean coverage from 82.59% to 82.90% across the ten configurations. The configuration-averaged recurrence-coverage gradient remained monotonic across all six levels. These results support the empirical stability of the Shared Core in the selected domain, but do not establish a universal core across unobserved implementations.

AuthorsOpenReview ↗

Looking forward to hearing from you

Dear Reviewer,

Thank you again for your time and constructive feedback. As the rebuttal deadline approaches, we would appreciate it if you could let us know whether our responses have addressed your concerns, or if you have any further questions. We would be happy to provide further clarification.

Thanks,

Authors

Thank you very much for the rebuttal! I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.