- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents BENCHPROOFER, a pipeline that takes a SWE task with a known patch, synthesizes formal specifications, translates or models relevant code in one of several verification backends (Nagini, Velvet, Lean), axiomatizes unchanged callees, and admits instances only after mechanical and LLM based adversarial checks. The authors applied this pipeline to SWE-bench Verified instances under multiple backend verifiers that yields SWE-PROOF, a benchmark containing real-world software-engineering issue resolution with formal verification rather than relying solely on hidden tests.
Critical strengths
An ambitious benchmark: This paper introduces an ambitious benchmark connecting real-world GitHub issue resolution with formal verification across multiple backends. Unlike other contemporary studies that works on verification-aware languages, this work targeted regular repository level SWE-task and attempts to add formal guarentes in those tasks. This benchmark also highlights important limitations of hidden-test evaluation on SWE-Bench style tasks using counterexample-oriented auditing.
An e2e Neuro-symbolic pipeline: Benchproofer is a layered admission pipeline whose components are explicitly mapped to the artifacts. It cleanly separates the (intent, specification), (axiom, callee), and (implementation, patch) trust boundaries, which makes the pipeline auditable at each step.
Critical weaknesses
I believe this work is potentially significant, but its central guarantees depend heavily on LLM generated specifications, code/proof translations, axioms, and LLM-based audits based on so many assumptions.
Problem Framing: The central guarantee ($ Verified \implies Resolve$) is weaker than claimed (Claim 1) because it relies on several non-formal, LLM-audited assumptions (axiom soundness, translation fidelity, specification-intent alignment) rather than pure formal proof. Even with proofs, is not guaranteed because of the specification's correctness. A weak/trivial specification can easily be verified, and the implementation can pass the tests and still fails in hidden tests or edge cases. Prior studies already found that and [2][3][5]. If the specification is incorrect, the implementation, proofs, lemmas that follows the specifications can be verified but still incorrect. The guarantee can only come from the specification correctness first not from mechanical verification. And this study already acknowledges that determining specification correctness is an open-ended question.
Specification Overfitting: Moreover, the benchmark construction pipeline fully relies on ground-truth patches and details extracted from hidden tests, which naturally introduces potential risks of specification leakage and overfitting even with mitigation gates in place. Becuase specifications may encode the reference implementation’s interpretation of the issue rather than the issue’s natural-language intent. Therefore, some task instances require relaxed disclosure because the hidden tests and reference patch contain behavior not present in the public issue. The authors already observed such cases.
Model and Agent Harness Diversity: As mentioned, the main experiments use Claude Opus 4.8, with a single run of four repetitions for each backend verifier. Also, it is mentioned the LLM-audit, specification synthesis, transformation are agents generated. However, the authors did not explicitly mention (in the main paper) which agents/ model they used to run the LLM-adversarial gates (axiom soundness, conformance, soundness/completeness, property suites, equivalence, leakage). If it is Claude code with Opus 4.8, then the output artifacts would be Claude-ish, as all the code translation, generation, adulting are from Claude. Also, the backed transformation (specifically Lean) requires a strong reasoning model that can write Lean. I believe the pipeline verified task will significantly drops using any other model that is not good at Lean like Sonnet or Gemini-3 or GPT-4 series. So, the pipeline is not model agnostic. The authors should acknowledge this.
Known Findings: I appreciate the authors efforts to take the challenges of repository level formally verified SWE-task generation, however, almost all of the findings/claim of this study have already been explored in prior studies.
- Findings 1: passing all tests does not establish correctness [1] (issues, known for decades)
- Findings 2: self-constructed specifications do not help (LLM can easily pass the verifier by writing a postcondition ensure true)[2]
- Findings 3 & 4: producing a correct specification is the bottleneck for verified code generation, heavily emphasized in [2][3][4]
- Findings 5: specification correctness tracks patch correctness (issues with spec overfitting) [5][6]
The major difference is that all these works (except [6]) identified the same findings in functional code snippets. And this works draws the same findings in repository level (which is kind of obvious)
- [1] Edsger W. Dijkstra “The Humble Programmer”
- [2] Evaluating LLM-driven User-Intent Formalization for Verification-Aware Languages.
- [3] Intent formalization: A grand challenge for reliable coding in the age of AI agents
- [4] Neuro-Formal Verification: Agentic Language-Agnostic Formal Program Reasoning
- [5] Spec-Harness: Measuring and Improving Behavioral Adequacy of LLM-Synthesized Formal Specifications
- [6] Verusage: A study of agent-based verification for rust systems
Questions for authors
-
[Q1] Which <model / agent harness> pairs were used in all LLM-adversarial gates (axiom soundness, conformance, soundness/completeness, property suites, equivalence, leakage)?
-
[Q2] How to distinguish an agent patch that is genuinely incorrect from one that intentionally differs from the ground-truth patch on behavior not specified by the issue?
-
[Q3] How often do human expert auditors agree with the LLM adversarial panels on specification faithfulness and equivalence, particularly for instances where the panel rejects a synthesized specification?
3 comments
Thank you for the detailed response. I appreciate the clarification on the models and agent harnesses, the additional GPT-5.5 results, and the human inspection. These address some of my concerns. However, my main concerns about specification correctness remain unresolved.
First, specification correctness: The authors describe several soundness and completeness gates, but these mainly rely on reference patches, tests, and LLM-based audits. While these checks can identify incorrect specifications, passing them does not guarantee that the specification captures the full intended behavior. The correctness claim still depends on assumptions about specification faithfulness, axiom soundness, and implementation equivalence.
Second, specification overfitting: The authors report that approximately 52% of the inspected specifications were more restrictive than human judgment, while also claiming no incorrect specifications were found. Although these observations are not necessarily contradictory, overly restrictive specifications may reject valid alternative patches. This reinforces my original concern about specification overfitting and false rejections.
Third, Claim 1 and the actual benefit of formal verification: My original concern about Claim 1 remains. Table 23 shows that 100% of agent-generated artifacts verify, but only around 90–92% resolve the issues. While this does not directly contradict the conditional Claim 1, it highlights that verification alone does not establish resolution when specification correctness is uncertain. Moreover, comparing Rows 5 and 7, providing ground-truth specifications improves Opus 4.8's pass rate by only 2.2–4.0 percentage points when localization and verification requirements are held fixed (though the gains are larger for GPT-5.5). This makes the reported improvement from 85% to approximately 95% less clear as evidence for the benefit of specifications alone. Since the final evaluation still depends on hidden tests, I am not convinced that the paper sufficiently demonstrates the additional correctness assurance provided by formal verification.
I recognize the value of extending formal verification to repository-level SWE tasks. However, these concerns remain important to the paper's central claims. Therefore, I will maintain my original recommendation.
Further Response to Reviewer Ku6r
Thank you for your response. We would like to clarify several points about specification correctness and the guarantees provided by SWE-Proof.
1. Specification correctness is extensively checked through mechanical and adversarial gates. Our methodology uses independent adversarial agents to actively falsify the correspondence between formal specifications and the intended task behavior. The soundness and completeness gates search for correct implementations that the specifications reject and incorrect implementations that they accept. The faithfulness audit specifically searches for required behavior from the original issue that the specifications fail to capture. The property-based falsification gate further challenges the specifications using both correct and incorrect implementations. Similarly, the axiom-soundness gate tests assumptions against actual repository code, while the conformance and equivalence gates check that the verified implementation corresponds to the actual Python patch. These checks are supported by extensive execution, including 16,118 property-based falsification cases and 129.8 million differential comparisons on SWE-bench Verified. All released instances pass the applicable correctness gates. We also manually inspected approximately 60 instances per verification backend and found no incorrect specifications. Together, these results provide strong empirical evidence that our specifications accurately capture the intended task behavior.
2. More restrictive specifications do not weaken the correctness guarantee. Our objective is to construct specifications that enforce correct downstream fixes. Some specifications may reject valid alternative implementations, but this does not affect the correctness of implementations that satisfy them. The reference implementation verifies against every released specification, demonstrating that a correct solution always exists. Coding agents can use feedback from the formal verifier to revise their implementations until they satisfy the specification. Our soundness and property-based falsification gates also explicitly check whether correct alternative implementations are unnecessarily rejected. Thus, stricter specifications may limit the set of accepted solutions, but they do not undermine our central claim that a verified implementation produces a correct fix.
3. Table 23 evaluates agent-generated specifications, not the specifications released in SWE-Proof. In Row 5, the agent receives only the location of the required changes and constructs its own specification. The fact that 100% of these implementations verify while only 90–92% resolve the issues demonstrates precisely the problem identified in our paper: agents can prove implementations against specifications that fail to capture the intended behavior. Claim 1 concerns verification against the specifications constructed and validated by Benchproofer, together with the correspondence between the verified implementation and the actual repository patch. With these specifications supplied in Row 7, verified resolution reaches 93.6–95.0% for Opus 4.8 and 94.4–94.6% for GPT-5.5.
4. Formal specifications provide both practical improvements and stronger correctness assurance. The comparison between Rows 5 and 7 isolates the benefit of supplying specifications when localization and verification requirements are held fixed. It shows improvements of 2.2–4.0 percentage points for Opus 4.8 and 6.4–8.6 points for GPT-5.5. The broader comparison from 85% to approximately 95% captures the combined benefits of formal specifications and their associated localization information, which our experiments explicitly separate. More importantly, formal verification provides correctness assurance beyond what finite test suites can establish. Our adversarial evaluation finds that 31.5% of Opus 4.8's and 58.9% of GPT-5.5's test-passing patches contain errors missed by the hidden tests. Formal verification establishes that an implementation satisfies its specification across the entire stated input domain, including inputs never exercised by tests. The hidden tests preserve comparability with SWE-bench, while formal verification provides an additional, machine-checked correctness criterion. This is the central contribution of SWE-Proof: enabling realistic coding agents to produce fixes supported by formal specifications and machine-checked proofs of correctness.
Response to Reviewer Ku6r
We thank Reviewer Ku6r for their thoughtful feedback. We answered your Q2, Q3 and other concerns in the response to all reviewers.
We use Opus 4.8 for all LLM calls but our approach is generalizable to use different frontier models. When constructing the SWE-Proof benchmark, we use Claude Code as the harness. When conducting experiments in Section 4, we use Opus 4.8 and a self-implemented harness for all settings within which some settings additionally have the verifier tool available. We additionally report the evaluation results on GPT-5.5 in Section 4 evaluated under the SWE-Proof instances generated by Opus 4.8. The results follow the same trend with Opus 4.8 and lead to the same conclusions in Section 4. We have incorporated these results into the revised manuscript.