acceptodds
Under review as a conference paper at ICLR 2027

SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

Abstract

Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check correctness with held-out test suites, which are inherently incomplete and increasingly susceptible to memorization. Formal verification avoids both problems, but existing work covers only standalone tasks whose specifications are given as input, not real issues, which touch large repositories and state intent in vague natural language. We present Benchproofer, a pipeline that turns a coding task with a known correct patch into a formally verified one: it writes a specification for the new code, summarizes the existing functions that code calls with axioms, and admits an instance only after mechanical and adversarial gates agree. Applying it to SWE-bench Verified yields SWE-Proof, 500 real issues whose correctness is formally verified rather than tested, and it extends to SWE-bench Pro. Evaluating Claude Opus 4.8, we find that verification catches what tests miss: a quarter of test-passing patches admit counterexamples, which a structured natural-language specification does not fix, while a correct formal one lifts resolution from 85% to 95%. Writing that specification is the hard part: an agent that must write its own gains nothing over an unaided baseline, and only 56% of those specifications pass our audit. The usual failure is faithfulness, a specification that constrains part of the required behavior and leaves the rest free. Specification quality still tracks the outcome, failing on 92% of unresolved instances against 51% of resolved ones, making faithful specification synthesis a concrete open problem.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Formal verification, Coding agents, Software engineering benchmarks, Specification synthesis, Verified code generation, Program repair

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ku6rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BENCHPROOFER, a pipeline that takes a SWE task with a known patch, synthesizes formal specifications, translates or models relevant code in one of several verification backends (Nagini, Velvet, Lean), axiomatizes unchanged callees, and admits instances only after mechanical and LLM based adversarial checks. The authors applied this pipeline to SWE-bench Verified instances under multiple backend verifiers that yields SWE-PROOF, a benchmark containing real-world software-engineering issue resolution with formal verification rather than relying solely on hidden tests.

Critical strengths

An ambitious benchmark: This paper introduces an ambitious benchmark connecting real-world GitHub issue resolution with formal verification across multiple backends. Unlike other contemporary studies that works on verification-aware languages, this work targeted regular repository level SWE-task and attempts to add formal guarentes in those tasks. This benchmark also highlights important limitations of hidden-test evaluation on SWE-Bench style tasks using counterexample-oriented auditing.

An e2e Neuro-symbolic pipeline: Benchproofer is a layered admission pipeline whose components are explicitly mapped to the artifacts. It cleanly separates the (intent, specification), (axiom, callee), and (implementation, patch) trust boundaries, which makes the pipeline auditable at each step.

Critical weaknesses

I believe this work is potentially significant, but its central guarantees depend heavily on LLM generated specifications, code/proof translations, axioms, and LLM-based audits based on so many assumptions.

Problem Framing: The central guarantee ($ Verified \implies Resolve$) is weaker than claimed (Claim 1) because it relies on several non-formal, LLM-audited assumptions (axiom soundness, translation fidelity, specification-intent alignment) rather than pure formal proof. Even with proofs, Verified  ⟹  ResolveVerified \implies Resolve is not guaranteed because of the specification's correctness. A weak/trivial specification can easily be verified, and the implementation can pass the tests and still fails in hidden tests or edge cases. Prior studies already found that Verified⇏ResolveVerified \nRightarrow Resolve and Verified⇏CorrectnessVerified \nRightarrow Correctness [2][3][5]. If the specification is incorrect, the implementation, proofs, lemmas that follows the specifications can be verified but still incorrect. The guarantee can only come from the specification correctness first not from mechanical verification. And this study already acknowledges that determining specification correctness is an open-ended question.

Specification Overfitting: Moreover, the benchmark construction pipeline fully relies on ground-truth patches and details extracted from hidden tests, which naturally introduces potential risks of specification leakage and overfitting even with mitigation gates in place. Becuase specifications may encode the reference implementation’s interpretation of the issue rather than the issue’s natural-language intent. Therefore, some task instances require relaxed disclosure because the hidden tests and reference patch contain behavior not present in the public issue. The authors already observed such cases.

Model and Agent Harness Diversity: As mentioned, the main experiments use Claude Opus 4.8, with a single run of four repetitions for each backend verifier. Also, it is mentioned the LLM-audit, specification synthesis, transformation are agents generated. However, the authors did not explicitly mention (in the main paper) which agents/ model they used to run the LLM-adversarial gates (axiom soundness, conformance, soundness/completeness, property suites, equivalence, leakage). If it is Claude code with Opus 4.8, then the output artifacts would be Claude-ish, as all the code translation, generation, adulting are from Claude. Also, the backed transformation (specifically Lean) requires a strong reasoning model that can write Lean. I believe the pipeline verified task will significantly drops using any other model that is not good at Lean like Sonnet or Gemini-3 or GPT-4 series. So, the pipeline is not model agnostic. The authors should acknowledge this.

Known Findings: I appreciate the authors efforts to take the challenges of repository level formally verified SWE-task generation, however, almost all of the findings/claim of this study have already been explored in prior studies.

  • Findings 1: passing all tests does not establish correctness [1] (issues, known for decades)
  • Findings 2: self-constructed specifications do not help (LLM can easily pass the verifier by writing a postcondition ensure true)[2]
  • Findings 3 & 4: producing a correct specification is the bottleneck for verified code generation, heavily emphasized in [2][3][4]
  • Findings 5: specification correctness tracks patch correctness (issues with spec overfitting) [5][6]

The major difference is that all these works (except [6]) identified the same findings in functional code snippets. And this works draws the same findings in repository level (which is kind of obvious)

Questions for authors
  • [Q1] Which <model / agent harness> pairs were used in all LLM-adversarial gates (axiom soundness, conformance, soundness/completeness, property suites, equivalence, leakage)?

  • [Q2] How to distinguish an agent patch that is genuinely incorrect from one that intentionally differs from the ground-truth patch on behavior not specified by the issue?

  • [Q3] How often do human expert auditors agree with the LLM adversarial panels on specification faithfulness and equivalence, particularly for instances where the panel rejects a synthesized specification?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ku6r

We thank Reviewer Ku6r for their thoughtful feedback. We answered your Q2, Q3 and other concerns in the response to all reviewers.

Model and harness diversity [Q1]

We use Opus 4.8 for all LLM calls but our approach is generalizable to use different frontier models. When constructing the SWE-Proof benchmark, we use Claude Code as the harness. When conducting experiments in Section 4, we use Opus 4.8 and a self-implemented harness for all settings within which some settings additionally have the verifier tool available. We additionally report the evaluation results on GPT-5.5 in Section 4 evaluated under the SWE-Proof instances generated by Opus 4.8. The results follow the same trend with Opus 4.8 and lead to the same conclusions in Section 4. We have incorporated these results into the revised manuscript.

SettingNaginiVelvetLeanEARS
0 Unaided baseline81.281.281.281.2
1 Baseline + Audit33.433.433.433.4
2 End-to-End80.280.681.081.4
3 Verified End-to-End80.479.479.632.2
4 Localization Provided87.087.087.087.0
5 Verified from Localization88.285.886.037.6
6 Specification Provided94.494.895.097.4
7 Verified from Specification94.694.494.451.4

Thank you for the detailed response. I appreciate the clarification on the models and agent harnesses, the additional GPT-5.5 results, and the human inspection. These address some of my concerns. However, my main concerns about specification correctness remain unresolved.

First, specification correctness: The authors describe several soundness and completeness gates, but these mainly rely on reference patches, tests, and LLM-based audits. While these checks can identify incorrect specifications, passing them does not guarantee that the specification captures the full intended behavior. The correctness claim still depends on assumptions about specification faithfulness, axiom soundness, and implementation equivalence.

Second, specification overfitting: The authors report that approximately 52% of the inspected specifications were more restrictive than human judgment, while also claiming no incorrect specifications were found. Although these observations are not necessarily contradictory, overly restrictive specifications may reject valid alternative patches. This reinforces my original concern about specification overfitting and false rejections.

Third, Claim 1 and the actual benefit of formal verification: My original concern about Claim 1 remains. Table 23 shows that 100% of agent-generated artifacts verify, but only around 90–92% resolve the issues. While this does not directly contradict the conditional Claim 1, it highlights that verification alone does not establish resolution when specification correctness is uncertain. Moreover, comparing Rows 5 and 7, providing ground-truth specifications improves Opus 4.8's pass rate by only 2.2–4.0 percentage points when localization and verification requirements are held fixed (though the gains are larger for GPT-5.5). This makes the reported improvement from 85% to approximately 95% less clear as evidence for the benefit of specifications alone. Since the final evaluation still depends on hidden tests, I am not convinced that the paper sufficiently demonstrates the additional correctness assurance provided by formal verification.

I recognize the value of extending formal verification to repository-level SWE tasks. However, these concerns remain important to the paper's central claims. Therefore, I will maintain my original recommendation.

AuthorsOpenReview ↗

Further Response to Reviewer Ku6r

Thank you for your response. We would like to clarify several points about specification correctness and the guarantees provided by SWE-Proof.

1. Specification correctness is extensively checked through mechanical and adversarial gates. Our methodology uses independent adversarial agents to actively falsify the correspondence between formal specifications and the intended task behavior. The soundness and completeness gates search for correct implementations that the specifications reject and incorrect implementations that they accept. The faithfulness audit specifically searches for required behavior from the original issue that the specifications fail to capture. The property-based falsification gate further challenges the specifications using both correct and incorrect implementations. Similarly, the axiom-soundness gate tests assumptions against actual repository code, while the conformance and equivalence gates check that the verified implementation corresponds to the actual Python patch. These checks are supported by extensive execution, including 16,118 property-based falsification cases and 129.8 million differential comparisons on SWE-bench Verified. All released instances pass the applicable correctness gates. We also manually inspected approximately 60 instances per verification backend and found no incorrect specifications. Together, these results provide strong empirical evidence that our specifications accurately capture the intended task behavior.

2. More restrictive specifications do not weaken the correctness guarantee. Our objective is to construct specifications that enforce correct downstream fixes. Some specifications may reject valid alternative implementations, but this does not affect the correctness of implementations that satisfy them. The reference implementation verifies against every released specification, demonstrating that a correct solution always exists. Coding agents can use feedback from the formal verifier to revise their implementations until they satisfy the specification. Our soundness and property-based falsification gates also explicitly check whether correct alternative implementations are unnecessarily rejected. Thus, stricter specifications may limit the set of accepted solutions, but they do not undermine our central claim that a verified implementation produces a correct fix.

3. Table 23 evaluates agent-generated specifications, not the specifications released in SWE-Proof. In Row 5, the agent receives only the location of the required changes and constructs its own specification. The fact that 100% of these implementations verify while only 90–92% resolve the issues demonstrates precisely the problem identified in our paper: agents can prove implementations against specifications that fail to capture the intended behavior. Claim 1 concerns verification against the specifications constructed and validated by Benchproofer, together with the correspondence between the verified implementation and the actual repository patch. With these specifications supplied in Row 7, verified resolution reaches 93.6–95.0% for Opus 4.8 and 94.4–94.6% for GPT-5.5.

4. Formal specifications provide both practical improvements and stronger correctness assurance. The comparison between Rows 5 and 7 isolates the benefit of supplying specifications when localization and verification requirements are held fixed. It shows improvements of 2.2–4.0 percentage points for Opus 4.8 and 6.4–8.6 points for GPT-5.5. The broader comparison from 85% to approximately 95% captures the combined benefits of formal specifications and their associated localization information, which our experiments explicitly separate. More importantly, formal verification provides correctness assurance beyond what finite test suites can establish. Our adversarial evaluation finds that 31.5% of Opus 4.8's and 58.9% of GPT-5.5's test-passing patches contain errors missed by the hidden tests. Formal verification establishes that an implementation satisfies its specification across the entire stated input domain, including inputs never exercised by tests. The hidden tests preserve comparability with SWE-bench, while formal verification provides an additional, machine-checked correctness criterion. This is the central contribution of SWE-Proof: enabling realistic coding agents to produce fixes supported by formal specifications and machine-checked proofs of correctness.

Reviewer_vFLTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BenchProofer, a pipeline for constructing formal specifications, reference implementations, and proofs for repository-level software repairs. It uses environment abstractions and mechanical and adversarial checks. The resulting SWE-Proof benchmark covers 500 SWE-bench Verified tasks and extends to the Python subset of SWE-bench Pro, using Nagini, Velvet, and Lean alongside structured natural-language requirements as a control. The experiments compare supplied and self-generated specifications and examine how faithfully the specifications capture the intended behavior.

Critical strengths
  1. Bringing verified code generation to real repository repairs is a valuable benchmark contribution. The three verification backends, explicit abstraction boundaries, executable adversarial cases, and extension to larger Pro tasks help explain where systems succeed or fail beyond what a test-pass score alone can show.

  2. The distinction between completing a proof and faithfully specifying the task is particularly useful. The worked examples and audits show how a valid proof can still leave important behavior unspecified. The localization controls and repeated Nagini runs add useful evidence to this analysis.

Critical weaknesses
  1. The different evaluation procedures make it difficult to assess the claim in Section 4.2 that only formal specifications close the correctness gap. Table 2 grades formal submissions using verification and an equivalence panel, while EARS and baseline row 1 use counterexample search. Appendix G.5 acknowledges that accepted formal submissions did not receive the same counterexample audit (p. 60, lines 3230–3233). This matters especially for row 3: an implementation is verified against a self-authored specification, but the later faithfulness audit finds many such specifications inadequate. The disclosed axiom and model-to-patch trust boundaries also need to be considered when interpreting correctness. A common audit on a prespecified matched sample of saved patches would help support the comparison; alternatively, the claim could be narrowed to what the current evaluations establish.

  2. Some discrepancies between the reported runs make the gain attributable to specifications hard to assess. Table 32 identifies r1 as the repetition reported in Table 2, yet localization-only is 91.4% versus 88.2%, and Nagini end-to-end is 84.2% versus 85.6%. The first difference affects the gain beyond localization. Page 53 also reports that 447 released specifications differ in at least one clause from the evaluated versions, although these changes may not be substantive. Identifying the evaluated snapshots and explaining how the runs map across tables would help resolve these questions. The main conclusions would also benefit from distinguishing the gains on Verified from the lower performance on Pro, already discussed in Appendix G.1: supplied formal specifications achieve 55.8–59.9%, versus 61.2% with localization alone.

Questions for authors
  1. Could the authors apply the same frozen counterexample-audit protocol and budget to a prespecified matched sample of saved patches from the main formal/EARS/unaided comparisons? The results would help determine how broadly the correctness-gap claim is supported.

  2. Which runs and specification snapshots underlie Tables 2, 3, and 32, and what accounts for the different values and any substantive revisions? For the test-pass audit, could the authors also explain how the 26.8-percentage-point drop in Table 2 relates to the conditional proportion of test-passing patches overturned and the 132/423 count in Appendix F.3?

  3. The AI-use statement says all manually inspected instances were correct. How many bundles were checked, how were they selected, and which aspects of specification faithfulness and model-to-patch correspondence were examined? These details would help explain what the manual inspection adds to the evidence from the inter-judge agreement.

For the discussion of evaluation design, AlphaDiana may be a useful comparison, particularly how it separates the agent, environment, and scorer interfaces and supports comparisons under shared conditions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer vFLT

We thank Reviewer vFLT for their thoughtful feedback. We answered your Q3 and other concerns in the response to all reviewers.

Apply the same frozen counterexample-audit protocol to three formal backends [Q1]

The counterexample audit for EARS in Table 2 uses the formal specifications from SWE-Proof as its reference. Specifically, an adversarial agent searches for inputs on which an EARS-generated implementation violates the formal specification. Each counterexample is then executed on both the generated patch and the ground-truth patch to confirm that it exposes an actual error. We also manually reviewed the counterexamples to ensure that the inputs satisfy the specifications' preconditions.

Formal verification provides a stronger guarantee than this counterexample search. When an implementation is formally verified against a specification, it is guaranteed to satisfy that specification for every input within its stated domain. Therefore, no counterexample of the kind searched for by the EARS audit can exist for an implementation verified against the same specification. In contrast, the EARS audit can only identify errors it successfully discovers; passing the audit does not establish that no errors remain. This explains why we use formal verification for the formal backends and counterexample search for EARS. Checking the three formally verified columns of Table 2 with the same counterexamples would result in an equal or higher pass rate than the one currently reported, never a lower one.

Run numbers don't match across tables [Q2]

Some results in the appendix, including Table 32, were evaluated on an earlier version of SWE-Proof. After these experiments, we added the conformance gate and re-audited the entire benchmark end-to-end with BenchProofer. Tables 2 and 3 report results on the updated benchmark, while Table 32 reports earlier evaluation runs, which explains the numerical differences. The specification revisions mentioned on page 53 also reflect updates made after the earlier evaluations. Importantly, both the earlier and updated results consistently support our main findings and do not change our conclusions. We have clarified these differences and the corresponding evaluation versions in the appendix of the revised manuscript to avoid confusion.

The 26.8-percentage-point drop in Table 2 is measured over all 500 benchmark instances, from 85.0% passing the hidden tests to 58.2% passing both the hidden tests and the adversarial audit. The corresponding conditional proportion of test-passing patches overturned is (85.0 − 58.2) / 85.0 = 31.5%. The 132/423 result in Appendix F.3 was obtained from an earlier evaluation on an older version of SWE-Proof, with a similar conditional overturn rate of 31.2%. We have clarified this distinction in the revised manuscript and corrected an incorrect statement in Section 4.2 that reported 26.8% as the proportion of test-passing patches overturned.

Comparison with AlphaDiana

Thank you for pointing us to AlphaDiana. AlphaDiana provides a general framework for evaluating agents by separating datasets, agents, environments, and scorers, enabling controlled comparisons across different models and agent frameworks. Our evaluation follows a similar principle by keeping the model, agent scaffold, task environment, and computational budget fixed while varying the information provided to the agent and the evaluation criteria. Unlike AlphaDiana, which focuses on general agent performance measurement, SWE-Proof focuses on constructing formal verification benchmarks for real-world software engineering tasks and evaluating the role of formal specifications and verification. We have cited and discussed AlphaDiana in the revised manuscript.

Thank you for the detailed responses. The version clarification, corrected overturn rate, and additional information on human inspection address several of my concerns. I will raise my recommendation in recognition of the benchmark and construction methodology. I would encourage the authors to clarify the scope of the guarantees, especially for self-authored specifications and the audited correspondence between verified implementations and repository patches. The finding that about 52% of inspected specifications were more restrictive than human judgment also deserves discussion, since it may affect which valid alternative fixes are accepted. These clarifications would help others interpret and build on this valuable contribution.

Reviewer_r2ATOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors present BenchProofer, a pipeline that adds formal verification to coding benchmarks. They apply it to SWE-Bench Verified, creating SWE-Proof.

Critical strengths

This is an interesting topic and field of work. The paper's evaluations is especially comprehensive and I appreciated section 3.5 that tries the author's methodology on a larger dataset.

Critical weaknesses

The authors claim: "Our experiments ask what verification is worth." But that is not what the authors measure. The point of formal verification is to check whether code satisfies a known-good specification. Instead, the authors derive the specification from bug reports. While the authors address the obvious challenges associated with getting LLMs to output good specs, this method does not lend itself to building a known-good spec from ground up. Rather, it is whack-a-mole with existing bugs, exactly the kind of working mode verification is meant to prevent. The claim "SWE-PROOF provides ground truth formal specifications, implementations, and proofs of correctness" appears overstated given that the authors' approach is only suitable to verifying what has once went wrong in bug reports.

While the authors use property-based testing as part of one of their "gates", they do not compare whether full formal verification is necessary or whether mere property-based tests derived from their specifications is performs similarly well. Overall, the paper tends to use the term "formal verification" fairly loosely.

Questions for authors

Can you clarify what makes you certain that the specifications (generated from bug reports) actually encompass the full task?

Have you tried property-based testing derived from your specs?

5 comments
AuthorsOpenReview ↗

Response to Reviewer r2AT

We thank Reviewer r2AT for their thoughtful feedback.

What makes us certain that the specifications (generated from bug reports) actually encompass the full task [Q1]

The key distinction is between verifying an implementation against a specification and ensuring that the specification faithfully captures the task's intended behavior. Our methodology explicitly addresses both.

First, our specifications are not generated from bug reports alone. During benchmark construction (Section 3.2), the agent examines the original issue, repository code, reference patch, and existing tests. These provide multiple sources of information about the intended behavior.

Second, our correctness gates go beyond checking whether the specification rejects the original bug. In particular:

  1. The soundness and completeness gates (Section 3.3) use independent adversarial agents to search for both correct implementations that the specification incorrectly rejects and incorrect implementations that it mistakenly accepts. These attacks are not restricted to the original bug or existing tests.
  2. The faithfulness audit (Section 3.3 and Appendix E.3) specifically searches for required behavior that the specification fails to capture. This is distinct from completeness: a specification may correctly constrain every function it models while omitting other behavior required by the issue.
  3. The property-based falsification gate (Section 3.3) generates additional correct and incorrect implementations, checking that the specification accepts or rejects them appropriately. The released SWE-Proof benchmark includes 16,118 such cases for SWE-Bench Verified.
  4. The conformance and equivalence gates check that the formally verified implementation corresponds to the actual repository patch. Our differential checks compare the modeled and actual repository behavior on 129.8 million generated inputs across SWE-Bench Verified.

Importantly, a failed check sends the specification back for revision. We do not simply add exceptions for previously discovered bugs. For example, Appendix H.2 presents a case where adversarial attacks revealed an entire class of incorrect implementations permitted by an earlier specification. We corrected the underlying model of database access, eliminating the class of errors rather than addressing individual test cases.

Our confidence in the correctness of the specifications also comes from the remarkable advances in frontier LLMs' ability to construct counterexamples, with the most capable systems now solving Millennium Prize-level mathematical problems. Although the models used in our work are less powerful, they are capable of identifying concrete counterexamples that expose errors and omissions in specifications for SWE-bench tasks. These recent advances in LLM capabilities have made it possible to apply formal verification to realistic software engineering tasks at a scale that previous work has not achieved.

Try property-based tests derived from the specifications in SWE-Proof [Q2]

The purpose of property-based testing is to build confidence in the correctness of an agent-written solution to a coding task when either (1) the specification of the problem is available only in natural language or (2) a formal specification is available but verifying that an implementation matches it is infeasible (e.g., because of solver run times).

Under (1), one of the primary contributions of our work is to construct correct formal specifications for the tasks in SWE-Proof, so we always have formal verification against which to compare. Importantly, property-based tests are also an integral part of our methodology. As described in Section 3.3, these tests are co-constructed with and informed by the formal specifications. They generate additional correct and incorrect implementations and check whether the specifications accept or reject them appropriately. This helps identify errors in the generated specifications and increases our confidence in their correctness.

Regarding (2), in our experiments, formally verifying that an implementation matches the specification was not the limiting factor; indeed, we produced proofs that the reference implementation matched the formal specification for every task in SWE-Proof. Thus, property-based testing informed by our constructed formal specifications would provide a weaker correctness guarantee than formal verification in Lean, Velvet, or Nagini, without addressing a performance bottleneck in our benchmark suite.

Thank you for engaging with this review.

I am not convinced by your response. You are ultimately relying on LLMs to write correct specs. While your instrumentation may help, there is still no guarantee for the correctness of your specs. Your comments on LLM strength at mathematics are misleading; being able to find counterexamples given a formal mathematical problem is unrelated to translating informal to formal specs. You also present no quantitative evidence that your specs are trustworthy.

Side note: the sentence "The purpose of property-based testing is to build confidence in the correctness of an agent-written solution to a coding task when either (1) the specification of the problem is available only in natural language" contains critical misunderstandings about property-based testing. If your problem is only specified in natural language, PBTs do not help you; they are formal and can usually be directly converted to verifiable propositions.

I must continue to recommend rejecting the paper.

AuthorsOpenReview ↗

Further Response to Reviewer r2AT

We thank Reviewer r2AT for the timely response and thoughtful feedback.

There is no mathematical guarantee that a formal spec faithfully captures an informal natural-language issue. Establishing such a guarantee would require an independently verified formalization of the original intent, which is precisely what realistic software engineering tasks lack. Our contribution is to construct useful and accurate formal specifications for these tasks and validate them through extensive mechanical and adversarial checks. This differs from traditional formal verification, where specifications are already given. More importantly, our experiments demonstrate the practical value of these specifications: supplying them improves resolution from 85.0% to 95.1% for Claude Opus 4.8 and from 81.2% to 94.7% for GPT-5.5, with substantial gains even after controlling for localization. Formal specifications are therefore useful for guiding real-world coding agents, even when their correspondence to informal intent cannot itself be mathematically proved.

The ability of LLMs to construct counterexamples is directly relevant to our methodology. We use independent adversarial agents to actively falsify the correspondence between the natural-language intent and the formal specification, as well as between the formally verified implementation and the actual repository patch. These agents search for incorrect implementations that satisfy the spec, correct implementations that it rejects, and concrete scenarios where the spec fails to capture the issue's intended behavior. Their findings are checked using the verifier, executable tests, and the actual repository. Repeated failures to construct such counterexamples provide strong evidence of specification correctness.

We also provide substantial quantitative evidence. On SWE-bench Verified, our methodology produces 16,118 property-based falsification cases and performs 129.8 million differential comparisons between modeled implementations and actual repository code. All released instances pass the applicable correctness gates. Furthermore, the same adversarial auditing methodology rejects 39–54% of specifications generated independently of BenchProofer, demonstrating its ability to identify incorrect specifications. We also manually inspected approximately 60 instances per formal verification backend, and all inspected specifications were correct. Some imposed stronger requirements than a human developer might consider necessary, but these requirements still enforce correct downstream fixes. These results provide concrete evidence of specification quality beyond relying on LLM-generated judgments alone. If there is another quantitative evaluation you believe is necessary, we would appreciate a specific suggestion, as the benchmark already includes extensive reproducible evidence from its mechanical and adversarial gates.

Regarding property-based testing, our PBTs are constructed to reflect the natural-language intent, including both correct behaviors that a specification should permit and incorrect behaviors that it should reject. These tests are expressed formally and used to identify specifications that are too strong or too weak. However, each test checks only a particular behavior or implementation, whereas formal verification establishes that an implementation satisfies the spec over its entire stated input domain. This is why we combine property-based falsification with formal verification rather than relying on testing alone. The tests help establish confidence that the specification captures the intended behavior, while formal verification enforces that specification on downstream implementations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank all reviewers for their careful feedback. We are encouraged that reviewers found the problem of bringing formal verification to real repository-level issues important and ambitious (r2AT, vFLT, Ku6r); found the multi-backend pipeline, with its explicit trust boundaries, auditable (vFLT, Ku6r); valued how it exposes the limits of test-based evaluation (vFLT, Ku6r); and found the evaluation comprehensive, including its extension to SWE-bench Pro (r2AT, vFLT).

Do the specifications capture the full intended behavior? Or are the generated specifications complete? (r2AT[Q1], vFLT, Ku6r)

The specification initially generated by an agent during a run of Benchproofer may not capture the full intended behavior of the natural language GitHub issue. This corresponds to the completeness of the spec, i.e., whether it rejects all the implementations that are not intended. However, our methodology takes several steps, including multiple correctness gates, to ensure that such initially incomplete specifications are corrected and the final generated specification is correct.

  1. In specification synthesis (Section 3.2) and the Completeness gate (Section 3.3), we ask the agent to adversarially propose an implementation that satisfies the specification but fails the tests or behaves differently from the reference implementation in a meaningful way. This step uses the LLM’s ability to make the specification cover more than what tests cover.
  2. In the Discrimination gate (Section 3.3), we make sure the spec rejects the pre-fix implementation.
  3. In the Mutation check gate (Section 3.3), we make sure that the spec rejects the first-order mutations with a high rejection rate.
  4. In the Property-based falsification gate (Section 3.3), we go beyond the test cases that existed in the repository and we generate additional property-based tests and implementations. We require the implementations that fail the property-based tests to be rejected by the specification, improving the completeness of the specification.

Specifications come from bug reports or reference patches, so they may fit the reference fix rather than the issue. Soundness of the specification. (r2AT, Ku6r[Q2])

We acknowledge that the specification initially generated by an LLM may overfit the reference fix and therefore reject an alternative fix whose behavior differs from the reference one only in ways that are orthogonal to the issue statement. This concern corresponds to the soundness of the spec, i.e., whether it can accept all the implementations that are intended. Whether a particular behavior is correct according to a natural language issue description is inherently ambiguous, but in our implementation, we provide multiple gates to ensure the soundness of the specification within this ambiguity:

  1. In the Soundness gate (Section 3.3), we apply a panel of adversarial agentic auditors to identify specifications that overfit the golden patch (lines 263-266) and use the results to broaden the specification until it is both sound and complete.
  2. In the Property-based falsification gate (Section 3.3), we go beyond the test cases that existed in the repository and generate additional property-based tests. These tests again check that the specification is broad enough to cover behavior outside the unit tests.

Reliance on LLM audits, with too little detail on human validation. (vFLT[Q3], Ku6r[Q3])

In our work, we constructed two verified benchmark sets, based on SWE-Bench Verified and SWE-Bench Pro. For these benchmarks, we (1) manually inspected approximately 60 instances for each backend during the development of our approach and (2) systematically reviewed 128 instances after completing the final construction of Benchproofer. The systematic inspection found no instances of incorrect specifications or agent cheating, but did identify approximately 52% cases in which the specification was more restrictive than human judgement as described above. During development our inspection identified many instances of incorrect specifications which we used to improve on the gates and construction methodology described in Section 3.

While constructing two correct benchmarks for verified code generation is a core part of our work, our most significant contribution is the general methodology for constructing a formally verified benchmark set, given an unverified set of tasks. The fact that this methodology is automated is one of its key advantages: it allows researchers or practitioners to generate a formally verified version of an arbitrary agentic coding task with high confidence provided by the combination of mechanical and agentic gates we implement. While some level of human review is still required for novel applications of Benchproofer, the fact that this validation is minimal is an advantage, not a cost, of our approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.