acceptodds
Under review as a conference paper at ICLR 2027

Self-Spec Verifiable Code Generation

Abstract

Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees. Recently, researchers have proposed several benchmarks to evaluate the capabilities of LLMs in generating formally verifiable code, where LLMs need to formulate formal specifications, generate the corresponding code, and verify its correctness. However, existing benchmarks have two key limitations: (I) They primarily evaluate specification and code generation stage-wise, with code generation typically conditioned on an oracle specification. This setup overlooks whether strong stage-wise performance translates into end-to-end success. (II) They mainly focus on a single proof-oriented language and mathematically structured tasks, offering limited coverage of tasks common in software development. In this paper, we introduce VeriCodeBench, a benchmark for self-spec verifiable code generation, where the LLM relies solely on its own generated specification and code throughout the entire process. VeriCodeBench contains 400 language-native problems across C, Java, Rust, and Python, covering practical concerns in software development. We evaluate specification coverage, code validity, and joint problem-level success across four representative LLMs. We further introduce CodeNova to enhance the capabilities of LLMs in self-spec verifiable code generation. CodeNova makes requirements explicit through constraint-guided specification and uses verifier feedback to guide targeted implementation repairs. Experimental results reveal that self-generated specifications remain a major bottleneck, while providing more detailed specifications may not necessarily lead to higher verification success rates. CodeNova substantially improves performance across all evaluation metrics, enabling Claude Sonnet 5 to achieve the strongest results under the challenging setting. Our code is anonymously available at https://anonymous.4open.science/r/ICLR27-VeriCodeBench .

Primary area:
datasets and benchmarks
Keywords:
Formal verification, verifiable code generation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ZfXUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces VERICODEBENCH, a new benchmark for self-spec verifiable code generation, requiring LLMs to create both formal specifications and code that is verified against the generated specifications. To address the gap where specification generation and code generation are typically evaluated separately, it covers 400 problems across four major languages. Additionally, the paper proposes CODENOVA, a framework that combines Constraint-Guided Specification (CGS) generation with Verifier-Guided Candidate Repair (VGCR) to improve end-to-end correctness. Experiments demonstrate that verifier acceptance alone often overestimates success, highlighting the critical dependency between generating accurate specifications and implementing verifiable code.

Critical strengths

E2E Verified Code Generation: The paper addresses an important and underexplored evaluation gap in end-to-end verified code generation when the model must rely on its own formal specification. The self-spec framing is a meaningful step toward realistic evaluation of verifiable code generation. The multilingual, native-toolchain benchmark with frozen-spec enforcement and released artifacts is also a useful resource.

Critical weaknesses

VERICODEBENCH Quality Issue: The authors describe the general dataset construction and quality control process, but they do not clearly explain the original source of the tasks or provide enough details about the curation process. They did not report the number of curators, whether obligations were independently reviewed, whether disagreements were measured, or how ambiguous natural-language requirements were resolved beyond reference-specification guidance.

As described, the benchmark covers multiple developer-facing languages and native verification ecosystems, exposing diverse proof obligations such as pointers, frames, ownership, permissions, and exceptions. However, looking into the artifacts, many tasks appear to be trivial and may not be a good representation of realistic SWE tasks considering the current capabilities of models/coding agents. Here is a sample of VERICODEBENCH tasks from frama-c-problems and java-problems. Similar types of tasks are also present in other languages.

  • Given an integer, return its absolute value.
  • Given a nonempty readable integer array, return its first element.
  • Given two integers, return the larger value.
  • Given two integers whose sum is within int range, return their sum.
  • Given a nonnegative integer n, iterate a counter from 0 up to n and return n.
  • Given a nonnegative integer n, return the arithmetic sum of integers from 0 to n inclusive.
  • Given a nonempty integer array and its length, return the maximum element.

These examples are too simple in terms of implementation, and their relevance to practical SWE tasks is unclear. Although simple implementations can still involve verification challenges, the authors have not provided enough evidence about the overall difficulty of the benchmark. All of these tasks are single-function implementations (many of them even just 1-line implementations). There are no proper statistics about the benchmark, such as median LOC, number of loops, branches, or overall complexity of tasks.

Most importantly, many of these tasks appear to require mainly preconditions, postconditions, and loop invariants, although the benchmark also includes frame conditions, memory safety, permissions, and exceptions. However, major challenges in verified code generation also include correct specification synthesis and complex auxiliary proof/lemma synthesis. Compared to VeriContest [1] (a benchmark for E2E Verified Code Generation), VeriCodeBench appears to have limited problem difficulty and does not provide the same explicit fine-grained task divisions (e.g., nl2spec, spec2code, nl_spec2code, etc.).

On the multi-language task contribution, I believe practical application matters more than language diversity alone. For instance, the paper does not provide evidence of real-world adoption or use cases for Java/JML and Python/Nagini. While these languages introduce different verification challenges, it is unclear how well the benchmark represents practical software engineering tasks.

CODENOVA's Novelty and Self-Spec Translation

CODENOVA combines constraint extraction, self-checking, specification refinement, verifier diagnostics, repair planning, and candidate generation. Its main components are CGS and VGCR, while CEF is used by VERICODEBENCH to evaluate specification coverage. CGS does the LLM-based specification translation and refinement, and CEF provides an automated and deterministic approximation of requirement-level adequacy. VGCR has already been explored in the literature through many LLM-assisted verification repair approaches [2][3] . Speaking of novelty, the new contributions of CGS and VGCR compared to prior work are not clearly established. CEF appears to be an important part of the benchmark contribution. I have some major concerns about CGS and CEF.

First, in VERICODEBENCH, the CGS component may benefit from the fact that the task NL requirements are already clear enough to identify what to verify and the edge cases. I believe CGS performance may significantly drop when translating and refining specifications for medium/hard difficulty tasks, for instance, a medium/hard task from VeriContest [1]. Is it possible to test this since CODENOVA can be run on Rust/Verus?

Moreover, CGS uses the same LLM to extract constraints, generate specifications, and perform self-checking. This creates a risk that the model may repeat its own mistakes during the self-check. The authors do not separately evaluate how often the self-check identifies missing or incorrect constraints. Therefore, it is unclear how much the self-check contributes to CGS's improvement.

However, that brings my second concern. The CEF evaluation requires manually curated reference constraints, which means CEF cannot directly validate arbitrary new requirements without trusted reference constraints. If the curated target set omits an important requirement, CEF cannot detect that omission. As described in the paper, curators sometimes use reference specifications to resolve implicit details in natural-language requirements. Therefore, evaluation depends partly on those curation decisions. Furthermore, CEF checks requirement coverage through logical entailment, but coverage does not necessarily guarantee that the full specification is correct, consistent, or free from unnecessary restrictions.

In the evaluation, the authors compared primarily against Direct, CGS-only, and VGCR-only variants. However, they do not compare against simpler verifier-feedback baselines such as iterative repair without structured planning or self-refinement without constraint extraction. More detailed ablations are needed to determine which parts actually contribute to the score. Also, VGCR and CODENOVA use many more LLM and verifier calls than Direct, but budgets (K, number of rounds, review budget) are not stated in the main text. A comparison under equal computation budgets is needed to understand whether the improvements come from the proposed methods or simply from additional attempts.

Questions for authors
  • [Q1] How were the 400 benchmark problems authored or selected, and were target obligations reviewed by multiple independent curators? Can the authors report task difficulty statistics such as LOC, number of loops and branches, and number of helper lemmas needed for verification?

  • [Q2] How does CEF encode quantifiers, sequence views, \old/final, pointer validity, and Nagini permissions into SMT? What fraction of target checks end in solver unknown/timeout or fall back to type-aware matching? Does CEF check for conflicting constraints or overly restrictive specifications?

  • [Q3] Can the authors evaluate CODENOVA on medium/hard Rust/Verus tasks from VeriContest [1] to demonstrate whether the approach generalizes to more difficult verification problems?

  • [Q4] What are the exact budgets of K candidates, repair rounds, self-check iterations, and verifier timeouts? How does CodeNova compare with a simple iterative verifier-guided repair baseline given the same number of LLM and verifier calls?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZfXU

Dear Reviewer ZfXU:

Thank you for the detailed feedback. We would like to clarify the benchmark’s intended scope, the role of CGS and CEF, and what the controlled experiments establish.

1. The benchmark studies the difficulty of completing the verification workflow.

The central question is whether a model can translate requirements into a formal specification and then produce an implementation that verifies against that specification, while preserving the intended behavior. The difficulty therefore includes decisions about admissible inputs, state changes, boundary cases, and the relationship between the specification and implementation. Implementation length alone does not capture these decisions.

Small functions provide a controlled setting in which these failures can be examined across verification ecosystems. We position this as complementary to benchmarks emphasizing harder algorithms and auxiliary proof synthesis, such as VeriContest. Our focus is the dependency between requirement formalization and downstream verification.

The motivation for multiple languages is similarly tied to verification semantics. Pointer validity and framing, JML state and exceptional behavior, Verus ownership and sequence views, and Nagini permissions impose different requirements on formalization and proof. Their inclusion allows us to investigate whether pipeline behavior varies with these obligations. The contribution does not depend on establishing widespread industrial adoption of every selected toolchain.

2. CEF is a benchmark evaluation procedure with a defined reference boundary.

CEF measures coverage relative to a fixed, curated set of requirement obligations. It is not intended to validate arbitrary new requirements without trusted targets. Reference obligations serve the same essential purpose as ground-truth artifacts in other benchmark settings: they provide an evaluation criterion independent of the candidate being assessed.

The distinction between the extracted constraints used by CGS and the curated targets used by CEF is important. The former are model-generated intermediate artifacts; the latter are withheld evaluation artifacts. Consequently, the generation process cannot directly optimize against the evaluator’s target clauses or feedback.

We also distinguish requirement coverage from complete semantic correctness. Coverage, specification consistency, input-domain admissibility, and native verifier acceptance are related but different properties. The motivation for reporting coverage alongside validity and joint success is to expose failures that verifier acceptance alone would conceal. CEF’s normalization, matching, and SMT checks should be understood within their supported scope, rather than as a general decision procedure for all constructs in the four specification languages.

3. The controlled comparisons provide evidence about pipeline behavior.

Appendix B.3 describes a useful control: VGCR reuses the exact specification and initial implementation from Direct, and CodeNova similarly reuses those from CGS. These artifacts are not resampled when repair is added. This helps distinguish repair outcomes from changes in the specification or initial code sample.

The comparisons therefore support an analysis of what happens when repair is added to a given starting point. Appendix C additionally examines sensitivity to sampling budget. These experiments address pipeline behavior; a comparison under equal computation budgets would address the separate question of efficiency.

Finally, we appreciate the requests for clearer provenance and curation reporting, structural difficulty statistics, finer component ablations, and explicit budget accounting. We acknowledge these limitations and intend to improve the documentation and broaden the validation in a future revision.

Thank you for the clarifications. However, I am not convinced about the benchmark's quality and difficulty, as many tasks are simple and may not reflect realistic verified code generation challenges. The rebuttal also does not fully address my concerns about CEF's reliability, CGS's effectiveness, and the lack of equal-budget comparisons. Therefore, I will maintain my rejection recommendation.

Reviewer_3G9bOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces VeriCodeBench, a 400-task benchmark across four langagues and formal systems that evaluates self-spec verifiable code generation. By self-spec verifiable code generation, it uses generated specificiation as the verification target instead of the ground truth specification. It also introduces CodeNova, a method combining constraint-guided specification and verifier-guided candidate repair to improve self-spec verifiable code generation performance.

Critical strengths

1/ The benchmark covers four different formal systems, which can show the difference between them and help understand the general difficulty of verifiable code generation

Critical weaknesses

1/ The importance of setting of "self-spec verifiable code generation" is overstated. I don't see how measuring self-spec at this paper's level contributes to understanding of verifiable code generation, for the following two reasons:

a) Self-spec can potentially trivialize formal verification, by using definitionally equivalent code/spec pairs, as already explained in VERINA [1] and others for example. The paper doesn't recognize this issue, and the soundness of the self-spec metric is in question. (for example, in Rust Problem006, code is x + 1 and spec is r == x + 1. For any tautology code/spec pair that mirrors each other, they may be valid if taken separately, but have no verification value).

b) The paper doesn’t study how spec and code interact. The evaluation still treats the two parts separately and just swaps the verification target to the self-generated spec. If the spec is bad, verifying code against it is meaningless, so why proceed with code generation at all? Therefore, it is unclear how to interpret the result of this paper and compare with the prior work in verifiable code generation.

2/ The evaluation and dataset scope is limited. This benchmark focuses on individual and rather simple functions. The baseline is a plain LLM call. The proposed CodeNova is also a combination of existing work, for example the idea of CGS (atomic requirement decomposition) can be found in VeriSpecGen [2] and others, and the idea of VGCR can be found in iterative refinemnet of VERINA [1] (which this paper directly compares to) and line of work that study how to leverage verifier feedback (e.g. Hilbert [3]). Repository verification and coding agent baselines are out of scope and not discussed in this paper.

[1] Verina: Benchmark Verifiable Code Generation. Ye et al. [2] Intent-aligned Formal Specification Synthesis via Traceable Refinement. Ye et al. [3] Hilbert: Recursively Building Formal Proofs with Informal Reasoning. Varambally et al.

Questions for authors

1/ Can authors clarify that in the current scope, is the tautology issue of self-spec considered, discussed, and handled during benchmark construction and evaluation? How the current benchmark construction and evaluation methodology prevent this issue?

2/ Can authors reason how the result and insight of this paper transfer to more realistic settings like large repositories and verification performed by coding agents?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3G9b

Dear Reviewer 3G9b:

We sincerely appreciate your thoughtful comments. We would like to clarify how the evaluation distinguishes specification adequacy from code–specification consistency, and how the experiments examine their interaction.

1. Self-spec does not make verifier acceptance sufficient for success.

We share your concern that a model can generate mutually consistent code and specifications that fail to capture the requested behavior. This concern motivates our distinction between requirement coverage and verifier acceptance. The generated specification is assessed against independently curated requirement obligations; it does not become the reference merely because the implementation satisfies it. Joint success requires both full requirement coverage under the evaluator and successful verification.

For the Rust example, the close correspondence between x + 1 and r == x + 1 does not, by itself, make the contract a logical tautology. Whether it is an adequate specification depends on the original requirement and its admissible input domain. A simple requested behavior may legitimately have a simple specification. The problematic case is a code–specification pair that agrees internally while omitting or misrepresenting the requested behavior.

The benchmark’s intended safeguard is therefore its independent, requirement-grounded assessment of specifications. We do not equate “verified against a self-generated specification” with “correct with respect to user intent.” Studying the gap between these outcomes is a central motivation for the setting.

2. The experiments explicitly examine specification–implementation interaction.

We respectfully clarify that the study goes beyond evaluating the two artifacts independently. Separate measurements allow us to distinguish failure modes, while joint success measures whether the complete workflow succeeds.

The four experimental configurations—Direct, CGS, VGCR, and CodeNova—examine interventions at specification generation, implementation repair, and their combination. Section 4.2 analyzes how improved specification coverage can introduce obligations that are harder for the implementation stage to satisfy. Conversely, verifier-guided repair can improve conformance to a fixed contract without recovering requirements missing from that contract. These observations concern the interaction between the stages.

Section 4.3 further compares self-generated and reference specifications to investigate how the specification supplied to the implementation stage affects downstream verification. This comparison changes both the available information and proof obligations, but provides a useful diagnostic of the specification’s role.

This also explains why proceeding with implementation remains informative when a generated specification is imperfect. It reveals whether a failure arises from formalizing the requirements, satisfying the resulting contract, or both. Filtering out imperfect specifications before implementation would obscure this behavior of an autonomous workflow.

We acknowledge the current limitations in task scale and baseline breadth, and intend to strengthen these aspects in a future revision.

Thank you for the response and I appreciate author's commitment to improve the manuscript in future revision. I believe the main concerns remain and is too substantial to be made within the rebuttal revision. Therefore I keep my rating of 1.

Reviewer_XPKVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces VeriCodeBench, a benchmark for self-spec verifiable code generation, where the model writes a formal specification from a natural-language requirement and then writes code that must verify against that same specification. The benchmark has 400 single-function problems, 100 each in 4 different languages. Specification coverage is measured with a Constraint Entailment Framework (CEF), which checks each manually curated target clause against subsets of the generated clauses with an SMT solver, and a problem is counted as success when all targets are covered and the code verifies. The paper also proposes CodeNova, a prompting pipeline that combines constraint-guided specification (CGS) with verifier-guided candidate repair (VGCR). Across four models, the paper reports 44.1% average joint success for direct generation and 49.1% for CodeNova, with Claude Sonnet 5 reaching 65.5%.

Critical strengths
  1. The paper tackles an important and timely problem of evaluating verifiable code generation end to end, where the model has to write both the specification and the code.
  2. Reporting multiple metrics is useful.
  3. The benchmark covers four languages with verifiers that are used in practice.
Critical weaknesses
  1. Tasks are mostly trivial: The abstract claims the problems cover "practical concerns in software development", but most are one or two line functions, eg "Given a boolean, return its negation" (Java #64). Only 15 of the 100 Java references have a loop, and Python has no loops or quantifiers. C problems 1-51 come from the public frama-c-problems exercises (with duplicates), which is not mentioned. Harder Java tasks such as is_sorted and binary_search were dropped when the reference did not verify (SKIPPED_UNVERIFIED in the released code).
  2. CEF is not validated and has clear errors: There is no check that CEF agrees with experts. In the released evaluator, only one generated precondition is checked, so adding requires \false does not lower coverage, which contradicts line 261. Quantified clauses cannot be parsed and fall back to text matching, so \result == a[i-1] covers \result == a[i+1], while 1 + \old(a[k]) does not cover \old(a[k]) + 1. A manual study on a sample of specifications would be helpful.
  3. CodeNova gains are small: CGS adds about 2.3 points on top of VGCR (49.1 vs 46.8), lowers joint success in 6 of 16 settings, and CodeNova is below Direct for Kimi (49.75 vs 51.25). All results are single runs, and CodeNova uses many more LLM calls than Direct with no budget-matched comparison. This makes the claim that it "substantially improves performance across all evaluation metrics" weak.

Other Weaknesses:

  1. C verification does not check memory safety: Frama-C is run without -wp-rte, so no goals are generated for invalid memory access or overflow, although these are listed as C-track obligations (Figure 1, Table 2). Eg the reference for problem 51 computes base * height with no overflow bound and still verifies.
  2. Missing prior work: VerifyThisBench already evaluates generating specifications, code and proofs together across Frama-C, VerCors and Verus, which covers both gaps claimed in the introduction. VeriEquivBench, Clover and VERINA's end-to-end setting are also not discussed.
Questions for authors

Please see the weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XPKV

Dear Reviewer XPKV:

We sincerely thank you for your detailed assessment and concrete examples. Below we address your concerns in detail:

1. Task difficulty and benchmark scope.

Our primary objective is to study whether models can independently carry out the complete process from natural-language requirements to formal specifications and verified implementations. In this setting, the model must determine what to specify, express those requirements formally, and produce an implementation that satisfies its own contract. Success therefore requires consistency across several dependent decisions.

We believe that implementation length and the presence of loops capture only part of this difficulty. Even a short function can expose a gap between satisfying a generated contract and adequately capturing the original requirements. The small tasks provide a controlled setting for examining this gap across different languages and verification tools. Nevertheless, we acknowledge that the tasks in our benchmark could be expanded in scale and complexity, and we plan to incorporate more challenging tasks in future revisions.

2. Contribution of CodeNova

The reported aggregate joint success improves from 44.1% with Direct to 49.1% with CodeNova. We recognize that the incremental gain over VGCR is smaller. However, the purpose of the study extends beyond establishing an aggregate performance gain.

Sections 4.2 and 4.3 specifically investigate the interaction between specification generation and implementation repair. Improving specification coverage can introduce obligations that are harder to satisfy, while successful repair against a fixed contract does not itself recover requirements omitted from that contract. These competing effects help explain why combining individually useful components does not always improve joint success.

The regressions are therefore relevant to the analysis: they expose when stronger specifications and verifier-guided repair do, or do not, work well together. We view these observations as a substantive outcome of the study, alongside the reported aggregate improvement. Our intended claim is that CodeNova provides a useful approach and experimental framework for understanding these dependencies, rather than that it dominates every model–language setting.

3.Relationship to prior work.

VERINA is explicitly discussed and compared with our work in the related-work section. We agree that its end-to-end setting is relevant. However, tasks are construct in a decoupled way. Our intended distinction concerns the particular evaluation protocol: generating a specification from the requirements, retaining that specification during implementation and repair, and separately examining requirement coverage and verifier acceptance.

We appreciate the reviewer’s concrete feedback on the evaluation methodology and verification scope. We acknowledge that the current study has limitations in these areas and intend to strengthen the validation and clarify the supported claims in a future revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.