- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces VERICODEBENCH, a new benchmark for self-spec verifiable code generation, requiring LLMs to create both formal specifications and code that is verified against the generated specifications. To address the gap where specification generation and code generation are typically evaluated separately, it covers 400 problems across four major languages. Additionally, the paper proposes CODENOVA, a framework that combines Constraint-Guided Specification (CGS) generation with Verifier-Guided Candidate Repair (VGCR) to improve end-to-end correctness. Experiments demonstrate that verifier acceptance alone often overestimates success, highlighting the critical dependency between generating accurate specifications and implementing verifiable code.
Critical strengths
E2E Verified Code Generation: The paper addresses an important and underexplored evaluation gap in end-to-end verified code generation when the model must rely on its own formal specification. The self-spec framing is a meaningful step toward realistic evaluation of verifiable code generation. The multilingual, native-toolchain benchmark with frozen-spec enforcement and released artifacts is also a useful resource.
Critical weaknesses
VERICODEBENCH Quality Issue: The authors describe the general dataset construction and quality control process, but they do not clearly explain the original source of the tasks or provide enough details about the curation process. They did not report the number of curators, whether obligations were independently reviewed, whether disagreements were measured, or how ambiguous natural-language requirements were resolved beyond reference-specification guidance.
As described, the benchmark covers multiple developer-facing languages and native verification ecosystems, exposing diverse proof obligations such as pointers, frames, ownership, permissions, and exceptions. However, looking into the artifacts, many tasks appear to be trivial and may not be a good representation of realistic SWE tasks considering the current capabilities of models/coding agents. Here is a sample of VERICODEBENCH tasks from frama-c-problems and java-problems. Similar types of tasks are also present in other languages.
- Given an integer, return its absolute value.
- Given a nonempty readable integer array, return its first element.
- Given two integers, return the larger value.
- Given two integers whose sum is within int range, return their sum.
- Given a nonnegative integer n, iterate a counter from 0 up to n and return n.
- Given a nonnegative integer n, return the arithmetic sum of integers from 0 to n inclusive.
- Given a nonempty integer array and its length, return the maximum element.
These examples are too simple in terms of implementation, and their relevance to practical SWE tasks is unclear. Although simple implementations can still involve verification challenges, the authors have not provided enough evidence about the overall difficulty of the benchmark. All of these tasks are single-function implementations (many of them even just 1-line implementations). There are no proper statistics about the benchmark, such as median LOC, number of loops, branches, or overall complexity of tasks.
Most importantly, many of these tasks appear to require mainly preconditions, postconditions, and loop invariants, although the benchmark also includes frame conditions, memory safety, permissions, and exceptions. However, major challenges in verified code generation also include correct specification synthesis and complex auxiliary proof/lemma synthesis. Compared to VeriContest [1] (a benchmark for E2E Verified Code Generation), VeriCodeBench appears to have limited problem difficulty and does not provide the same explicit fine-grained task divisions (e.g., nl2spec, spec2code, nl_spec2code, etc.).
On the multi-language task contribution, I believe practical application matters more than language diversity alone. For instance, the paper does not provide evidence of real-world adoption or use cases for Java/JML and Python/Nagini. While these languages introduce different verification challenges, it is unclear how well the benchmark represents practical software engineering tasks.
CODENOVA's Novelty and Self-Spec Translation
CODENOVA combines constraint extraction, self-checking, specification refinement, verifier diagnostics, repair planning, and candidate generation. Its main components are CGS and VGCR, while CEF is used by VERICODEBENCH to evaluate specification coverage. CGS does the LLM-based specification translation and refinement, and CEF provides an automated and deterministic approximation of requirement-level adequacy. VGCR has already been explored in the literature through many LLM-assisted verification repair approaches [2][3] . Speaking of novelty, the new contributions of CGS and VGCR compared to prior work are not clearly established. CEF appears to be an important part of the benchmark contribution. I have some major concerns about CGS and CEF.
First, in VERICODEBENCH, the CGS component may benefit from the fact that the task NL requirements are already clear enough to identify what to verify and the edge cases. I believe CGS performance may significantly drop when translating and refining specifications for medium/hard difficulty tasks, for instance, a medium/hard task from VeriContest [1]. Is it possible to test this since CODENOVA can be run on Rust/Verus?
Moreover, CGS uses the same LLM to extract constraints, generate specifications, and perform self-checking. This creates a risk that the model may repeat its own mistakes during the self-check. The authors do not separately evaluate how often the self-check identifies missing or incorrect constraints. Therefore, it is unclear how much the self-check contributes to CGS's improvement.
However, that brings my second concern. The CEF evaluation requires manually curated reference constraints, which means CEF cannot directly validate arbitrary new requirements without trusted reference constraints. If the curated target set omits an important requirement, CEF cannot detect that omission. As described in the paper, curators sometimes use reference specifications to resolve implicit details in natural-language requirements. Therefore, evaluation depends partly on those curation decisions. Furthermore, CEF checks requirement coverage through logical entailment, but coverage does not necessarily guarantee that the full specification is correct, consistent, or free from unnecessary restrictions.
In the evaluation, the authors compared primarily against Direct, CGS-only, and VGCR-only variants. However, they do not compare against simpler verifier-feedback baselines such as iterative repair without structured planning or self-refinement without constraint extraction. More detailed ablations are needed to determine which parts actually contribute to the score. Also, VGCR and CODENOVA use many more LLM and verifier calls than Direct, but budgets (K, number of rounds, review budget) are not stated in the main text. A comparison under equal computation budgets is needed to understand whether the improvements come from the proposed methods or simply from additional attempts.
Questions for authors
-
[Q1] How were the 400 benchmark problems authored or selected, and were target obligations reviewed by multiple independent curators? Can the authors report task difficulty statistics such as LOC, number of loops and branches, and number of helper lemmas needed for verification?
-
[Q2] How does CEF encode quantifiers, sequence views,
\old/final, pointer validity, and Nagini permissions into SMT? What fraction of target checks end in solverunknown/timeout or fall back to type-aware matching? Does CEF check for conflicting constraints or overly restrictive specifications? -
[Q3] Can the authors evaluate CODENOVA on medium/hard Rust/Verus tasks from VeriContest [1] to demonstrate whether the approach generalizes to more difficult verification problems?
-
[Q4] What are the exact budgets of K candidates, repair rounds, self-check iterations, and verifier timeouts? How does CodeNova compare with a simple iterative verifier-guided repair baseline given the same number of LLM and verifier calls?
2 comments
Thank you for the clarifications. However, I am not convinced about the benchmark's quality and difficulty, as many tasks are simple and may not reflect realistic verified code generation challenges. The rebuttal also does not fully address my concerns about CEF's reliability, CGS's effectiveness, and the lack of equal-budget comparisons. Therefore, I will maintain my rejection recommendation.
Response to Reviewer ZfXU
Dear Reviewer ZfXU:
Thank you for the detailed feedback. We would like to clarify the benchmark’s intended scope, the role of CGS and CEF, and what the controlled experiments establish.
1. The benchmark studies the difficulty of completing the verification workflow.
The central question is whether a model can translate requirements into a formal specification and then produce an implementation that verifies against that specification, while preserving the intended behavior. The difficulty therefore includes decisions about admissible inputs, state changes, boundary cases, and the relationship between the specification and implementation. Implementation length alone does not capture these decisions.
Small functions provide a controlled setting in which these failures can be examined across verification ecosystems. We position this as complementary to benchmarks emphasizing harder algorithms and auxiliary proof synthesis, such as VeriContest. Our focus is the dependency between requirement formalization and downstream verification.
The motivation for multiple languages is similarly tied to verification semantics. Pointer validity and framing, JML state and exceptional behavior, Verus ownership and sequence views, and Nagini permissions impose different requirements on formalization and proof. Their inclusion allows us to investigate whether pipeline behavior varies with these obligations. The contribution does not depend on establishing widespread industrial adoption of every selected toolchain.
2. CEF is a benchmark evaluation procedure with a defined reference boundary.
CEF measures coverage relative to a fixed, curated set of requirement obligations. It is not intended to validate arbitrary new requirements without trusted targets. Reference obligations serve the same essential purpose as ground-truth artifacts in other benchmark settings: they provide an evaluation criterion independent of the candidate being assessed.
The distinction between the extracted constraints used by CGS and the curated targets used by CEF is important. The former are model-generated intermediate artifacts; the latter are withheld evaluation artifacts. Consequently, the generation process cannot directly optimize against the evaluator’s target clauses or feedback.
We also distinguish requirement coverage from complete semantic correctness. Coverage, specification consistency, input-domain admissibility, and native verifier acceptance are related but different properties. The motivation for reporting coverage alongside validity and joint success is to expose failures that verifier acceptance alone would conceal. CEF’s normalization, matching, and SMT checks should be understood within their supported scope, rather than as a general decision procedure for all constructs in the four specification languages.
3. The controlled comparisons provide evidence about pipeline behavior.
Appendix B.3 describes a useful control: VGCR reuses the exact specification and initial implementation from Direct, and CodeNova similarly reuses those from CGS. These artifacts are not resampled when repair is added. This helps distinguish repair outcomes from changes in the specification or initial code sample.
The comparisons therefore support an analysis of what happens when repair is added to a given starting point. Appendix C additionally examines sensitivity to sampling budget. These experiments address pipeline behavior; a comparison under equal computation budgets would address the separate question of efficiency.
Finally, we appreciate the requests for clearer provenance and curation reporting, structural difficulty statistics, finer component ablations, and explicit budget accounting. We acknowledge these limitations and intend to improve the documentation and broaden the validation in a future revision.