- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents PV-Bench, a benchmark for evaluating LLMs on the formal specification and verification of fixed C programs. The benchmark consists of 233 programs with expert-reviewed reference specifications, annotations, and proofs. Using QCP and Rocq, PV-Bench evaluates three tasks: specification generation, annotation generation and proof generation. For specification evaluation, the authors introduce a proof-based judging mechanism that checks semantic equivalence against reference specifications and categorizes incorrect specifications as too weak, too strong, or incomparable. Experiments on frontier LLMs show that specification and annotation generation remain challenging, with the best models achieving pass@1 rates of 39.3% and 20.0%, respectively, compared to 65.5% for proof completion. The papers concludes that writing specifications and annotations is the main bottleneck in verification.
Critical strengths
-
Challenging benchmark with comprehensive verification artifacts. PV-Bench provides 233 fixed C programs, including competition-level algorithmic problems, accompanied by expert-reviewed specifications, annotations, and verification proofs. The inclusion of explicit memory reasoning and nontrivial algorithms introduces additional verification challenges beyond those typically covered by safe Rust-based benchmarks.
-
Fine-grained evaluation of verification capabilities. The benchmark separately evaluates specification generation, annotation generation, and proof completion, enabling a more detailed analysis of LLM verification capabilities. Its proof-based specification evaluation also provides a mechanism for assessing semantic equivalence and identifying different types of specification errors.
Critical weaknesses
W1: Limited novelty and insufficiently justified motivation. The benchmark largely combines evaluation tasks and methodologies already explored in prior work, including VERINA [1], VeriContest [2], and LiveFMBench [3]. Moreover, the motivation overstates the limitations of existing benchmarks. For example, the authors suggest that joint specification and code generation can lead to degenerate solutions, overlooking the independent correctness checks employed by VERINA and VeriContest. While extending verification benchmarks to C programs with explicit memory reasoning is valuable, the paper does not sufficiently distinguish its methodological contributions from prior work or justify the need for a separate benchmark.
W2: Insufficient experimental evaluation and limited discussion. The experimental evaluation is limited in breadth and depth, with relatively few models and insufficient analysis of benchmark reliability. In particular, T2 evaluation depends on an LLM judge successfully proving the generated verification conditions, potentially conflating annotation quality with the judge's proving capabilities. The paper does not adequately quantify this effect or report unresolved judgment rates. Moreover, the paper provides insufficient discussion of key benchmark-quality concerns, including dataset distribution diversity, potential data contamination, and threats to validity. In particular, although the reference specifications are reportedly manually reviewed, the authors do not clarify the review criteria, number of reviewers, or procedures for handling disagreements and potential errors. These limitations weaken the reliability and comprehensiveness of the evaluation.
[1] Ye, Zhe, et al. Verina: Benchmarking verifiable code generation. 2026.
[2] Xie, Zichen, et al. VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation. 2026.
[3] Xu, Dong, et al. LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation. 2026.
Questions for authors
Q1: Novelty and motivation. What fundamentally new evaluation capabilities does PV-Bench provide beyond existing benchmarks such as VERINA, VeriContest, and LiveFMBench? In particular, the authors argue that joint specification and code generation may lead to degenerate solutions. However, both VERINA and VeriContest independently check the correctness of generated specifications. How does the authors' concern apply to these benchmarks?
Q2: Reliability of the judging process. Since T2 evaluation depends on an LLM judge successfully proving the generated verification conditions, could the authors report the proportions of refuted and undecided submissions for each model? Additionally, how reliably can the judge verify known-correct annotations, and how sensitive are the reported success rates to the judge's proving capabilities?
Q3: Validity of reference specifications. The paper provides insufficient details about how reference specifications are manually reviewed. How many reviewers are involved, what review protocol is followed, and how are disagreements resolved? Since manual review cannot guarantee correctness, how do the authors assess potential errors in reference specifications and their impact on benchmark validity? Moreover, Section 2 says specifications "should state this property without describing how the program computes it", but the boundary between mathematical definitions and implementation details is unclear for complex DP and greedy problems. For example, Climbing Stairs can be specified using a mathematical recurrence, while greedy problems may have multiple equally valid solutions. how do the authors ensure that reference specifications capture the intended problem behavior rather than the specific algorithms used in their implementations? Additionally, Appendix C.2 appears to impose an implementation-specific output-buffer constraint not required by the original problem. Could the authors clarify this discrepancy?
Q4: Experimental coverage and reliability. Why is annotation generation evaluated on only 50 programs, with different subsets of models evaluated across tasks? Moreover, the evaluation covers only a limited number of proprietary models, lacking diversity in model families and architectures. Could the authors expand the evaluation to include more models, particularly open-source models, and analyze how verification performance varies with model size? A broader evaluation would provide stronger evidence of the benchmark's generalizability and enable more comprehensive insights into LLM verification capabilities.
Q5: Dataset diversity and contamination. Given that a substantial portion of the benchmark programs are generated by only two LLMs, how do the authors assess potential biases or limited distribution diversity in the benchmark? Additionally, there is no data contamination discussion since the benchmark contains publicly available competitive-programming problems and classical algorithms. Can the authors provide such discussions?
10 comments
Response(2/2)
Q4. Experimental coverage and reliability
All 223 programs have complete materials for all three tasks. T2 is evaluated on 50 programs because it is the most expensive task: each submission needs a generation session and a separate judge session, so we evaluated a stratified sample. T1 no longer uses a subset. The rerun in point 2 of the general response covers all 223 programs, including the 12 graph programs previously left out, for each model. The models also show how performance varies with model size within a family: Equivalent specifications rise from 98 (Luna) to 115 (Terra) to 118 (Sol) for GPT-5.6, and from 116 (Sonnet 5) to 136 (Opus 5) for Claude. We also evaluated an open-source model, Hy4. In T1 it produces Equivalent specifications for 105 of the 223 programs (47.1%), and in T3 it completes the proofs for 112 of the 223 programs (50.2%). Its T2 evaluation is in progress.
Q5. Dataset diversity and contamination
Only the 147 Codeforces programs are model-generated. Each follows the problem's standard solution, so the diversity comes from the problems, which span six algorithm families and ratings from 800 to 2800. The programs may appear in training data, but PV-Bench does not evaluate whether a model can solve these problems or write their solutions. The program is given and the model must specify and verify it, so having seen a solution in training gives little advantage.
how PV-Bench judges specifications with memory reasoning (supplement to Q1)
For specifications, prior work compares preconditions and postconditions separately, as implications between propositions over values, and neither method can compare function specifications that involve memory reasoning. Let be the ground-truth precondition and postcondition and the generated ones, over inputs and output .
VERINA checks four implications, by proof first and by tests otherwise:
VeriContest proves the precondition equivalence and tests the postcondition on correct pairs and mutated pairs :
With explicit memory reasoning, comparing preconditions and postconditions separately is no longer correct. A precondition also states which memory the function owns, the two conditions share logical variables quantified over the whole contract, and the postcondition must be related to the memory the precondition left untouched. For example, let require and return an array of cells, and one of cells. Neither precondition implies the other, yet every implementation of also meets , because the extra cell is left untouched (Section 4.2).
PV-Bench judges a candidate against the reference by function specification derivation, one entailment per direction, over the logical variables and of the two contracts (Section 4.2):
S\Rightarrow \hat{S}:&\quad \forall y.\ \hat{P}(y) \vdash \exists x.\ P(x) * \bigl(Q(x) \mathrel{-\!\!*} \hat{Q}(y)\bigr)\\ \hat{S}\Rightarrow S:&\quad \forall x.\ P(x) \vdash \exists y.\ \hat{P}(y) * \bigl(\hat{Q}(y) \mathrel{-\!\!*} Q(x)\bigr) \end{aligned}$$ A caller holding $\hat{P}$ splits off $P$ for the call, and the wand returns the rest to give $\hat{Q}$. Each direction is proved in Rocq or refuted by a Rocq-checked counterexample on a concrete memory state, giving the Equivalent, Too Weak, Too Strong, and Incomparable verdicts. A candidate is accepted only on proof, and what the judge cannot settle is reported as Undecided. **Example: two specifications that differ only in the memory they own.** In Codeforces 1139B, `main` declares a buffer larger than any single input uses, and `solver` reads only its first `n` cells: ```c static long long solver(const int *a, int n) { /* reads a[n-1] down to a[0] */ } int main(void) { int n; scanf("%d", &n); static int a[200005]; for (int i = 0; i < n; i++) scanf("%d", &a[i]); printf("%lld\n", solver(a, n)); } ``` The reference precondition owns the cells that `solver` reads. The precondition written by Sonnet 5 also owns the unused tail of the buffer: ``` Reference P : 1 <= n <= 200000 && IntArray::full(a, n, l) Sonnet 5 P̂ : 1 <= n <= 200000 && IntArray::full(a, n, l) * IntArray::undef_seg(a, n, 200005) ``` **PV-Bench judges Sonnet's specification Too Weak, where VERINA and VeriContest would accept it as equivalent.** The two specifications state the same propositions over values: the bounds are the same, and the judge proved the output relations equivalent. They differ only in the memory they own, which the derivation detects: - $S\Rightarrow\hat{S}$ is proved. A caller holding $\hat{P}$ passes the first `n` cells to the call and keeps the tail as the frame. - $\hat{S}\Rightarrow S$ is refuted by a Rocq-checked memory state with `n = 1` that owns the one array cell required by $P$ but not the next cell, which $\hat{P}$ also requires. A caller that owns exactly `n` cells therefore cannot call a function with Sonnet's specification. The derivation generalizes the pure criteria. When the assertions mention no memory and the logical variables are just the inputs, * becomes $\land$ and $-*$ becomes $\Rightarrow$, and $\hat{S}\Rightarrow S$ reduces to $$\forall x.\ P(x) \Rightarrow \hat{P}(x) \quad\text{and}\quad \forall x, r.\ P(x) \land \hat{Q}(x,r) \Rightarrow Q(x,r),$$ which are exactly VERINA's two soundness conditions. In the same way, $S\Rightarrow\hat{S}$ reduces to $$\forall x.\ \hat{P}(x) \Rightarrow P(x) \quad\text{and}\quad \forall x, r.\ \hat{P}(x) \land Q(x,r) \Rightarrow \hat{Q}(x,r).$$ The first is VERINA's completeness of $\hat{P}$. The second is VERINA's completeness of $\hat{Q}$ with the assumption $P(x)$ replaced by $\hat{P}(x)$: a candidate that accepts fewer inputs is compared with the reference only on the inputs it accepts. Since the first condition gives $\hat{P} \Rightarrow P$, VERINA's condition implies ours, and the two coincide whenever $P \Rightarrow \hat{P}$ also holds. In particular, a pure candidate is judged Equivalent, with both directions proved, exactly when it satisfies all four of VERINA's conditions.
Response(1/2)
We thank the reviewer for the careful reading. Our general response describes what PV-Bench adds and what we changed; here we answer this review's points.
Q1. Novelty and motivation
The general response clarifies our main contribution: explicit memory reasoning and harder problems. Among the benchmarks the reviewer names, VERINA works on immutable Lean values and VeriContest on safe Rust, so neither asks a model to reason about pointers and aliasing. LiveFMBench does target C, but it relies on automated solvers, so it cannot support competition-level algorithms and does not evaluate proving verification conditions.
Our concern about degenerate solutions is about who writes the program. VERINA and VeriContest do check generated specifications, but when the model also writes the code, it may pick an algorithm that is easy to verify. In VeriContest's October 2026 End2End release, 35 of the 107 problems with a fully correct answer have an accepted answer that does not implement the reference algorithm. PV-Bench does not let the model that verifies a program write its code: the program is given and fixed in all three tasks.
Memory also changes how specifications must be judged, so PV-Bench introduces a different judging method. The follow-up comment explains it.
Q2. Reliability of the judging process
Point 3 of the general response gives the proportions of accepted, refuted, undecided (Unresolved), and invalid submissions for each model, under pass@1 and the improved pass@2 judging.
For known-correct annotations, the judge is given the reference annotations, their verification conditions, and the Rocq proofs of those conditions (Section 4.1). With these, it can complete the proofs for the reference annotations of all 50 programs within the time limit. For correct annotations that differ from the reference, we believe the improved pass@2 judging is reliable. The three models' submissions contain 53 correct annotations in total. After the one-hour pass@2, 48 of them are verified (23 in pass@1 and 25 more in pass@2), and extending the second limit to three hours verified the other 5. Human review of the submissions still undecided found no further correct annotation.
Regarding how sensitive the reported success rates are to the judge's proving capabilities: even a weak judge gives a sound range for the success rate. Invalid submissions are detected mechanically and need no proving capability. Every Accepted verdict carries a Rocq-checked proof and every Refuted verdict a Rocq-checked counterexample, so the accepted submissions give a lower bound on the success rate, and the accepted plus undecided ones give an upper bound. A weaker judge only widens this range; it cannot make either bound wrong. So the range a weaker judge gives remains useful, and even the partial proof scripts it leaves on undecided submissions can support further analysis.
Q3. Validity of reference specifications
Two human experts review each reference against the problem statement, and it enters the benchmark only after passing both. They apply three criteria: the specification is independent of the algorithm; it admits every result the problem allows; and it states the mathematical result and the memory resources on both the success and the failure path. So Climbing Stairs is specified by counting step sequences, not by the recurrence, and an optimization problem by the optimum over the valid solutions. The reviewer is right about Appendix C.2; points 1 and 2 of the general response describe the correction, the audit of input ranges, and the T1 rerun.