acceptodds
Under review as a conference paper at ICLR 2027

Verified but Vacuous: Three Failure Surfaces of LLM-Driven Verified Code Generation

Abstract

What should “verified” mean when evaluating LLM-generated code? Many existing pipelines report a single verification number (for example, dafny verify accepted N% of samples). We argue this collapses three properties that are logically nested and empirically separable: verifier acceptance (Surface A), syntactic specification binding (Surface B: does the proof obligation bind to the spec?), and semantic specification fidelity (Surface C: does the spec capture the informal requirement?). We contribute this three-surface decomposition plus the ensures-binding gate, a syntactic Surface-B instance for Dafny pipelines that emit distinguishable specification and implementation artifacts (static AST walk, ms per component). Two empirical demonstrations show the surfaces separate. First (), via Li@Opus (a matched-model reproduction of li2025dafny's Dafny-as-intermediate-representation (IR) pipeline under Claude Opus 4.7) on three Python benchmarks of simple coding tasks (LeetCode-style): it reaches 99.1% verifier acceptance on HumanEval while only 1.9% carry a certificate binding to a spec-defined predicate or function. This gap is roughly fortyfold ( to ) that holds across HumanEval, LeetCodeDataset, and LiveCodeBench v6 in a narrow 1.9% to 2.7% band. This gap is observed even as the fraction of verified samples that carry any ensures clause climbs from 5.3% on HumanEval to 10.9% on LiveCodeBench v6. Postcondition prevalence more than doubles with benchmark difficulty but binding-to-a-spec-predicate rate does not follow. Second (), via VCG on two Rust benchmarks: we use VCG (which is our own in-house Dafny-as-intermediate pipeline) on Rust because the Rust target lets us enforce fully-verifiable output (every component ships as a Dafny certificate, ensures-binding gate passing by construction). Even inside that construction, 11.4% (MBPP-Rust) and 14.82% (VeriContest LC slice) of structurally-verified samples still fail hidden tests. On the LC slice of VeriContest, VCG's End2End rate is 7.79% (535/6870) at pass@1, +2.3 pp above the best previously published End2End LC number (GPT-5.5's 5.51%) and 4.14 the matched-backend single-shot Opus 4.7 End2End LC baseline (1.88%) that the VeriContest authors report per-slice. Therefore, any pipeline shipping certificates should report at least one of Surface B (e.g. the ensures-binding gate) or Surface C (e.g. Post2Exe) alongside Surface A; where feasible, both, since Surface B alone leaves the semantic blind spot we demonstrate on VCG, and Surface C alone leaves the syntactic blind spot we demonstrate on Li@Opus. Many existing evaluations report neither.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM code generation, verification failure

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_AuMyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that the verifier pass rate overstates what LLM-generated code actually verifies, since verified code may have no postcondition or a wrong specification. It proposes a static Dafny check that requires a postcondition to reference a predicate in the spec file. The check is applied to a reproduction of Li et al.'s pipeline and to the authors' own pipeline.

Critical strengths

1/ The reproduction of Li et al.'s pipeline is a concrete example that verifier acceptance alone can be misleading.

Critical weaknesses

1/ The contribution is overstated. Existing work already evaluates spec quality directly. VERINA [1] checks soundness and completeness against reference specs, CLEVER [2] proves equivalence to reference specs to rule out vacuous ones. The proposed check doesn't add much on top of these. A missing postcondition is just the weakest spec, and whether a postcondition names a predicate from another file is a writing convention, not a measure of meaning. An equivalent inline postcondition fails the check, while a tautology that mentions the predicate passes it.

2/ The evaluation is very hard to follow. The paper defines three surfaces (A, B, C), but the Li et al. results are reported in three levels that don't match them. Level 2 (any postcondition) has no corresponding surface, and Level 3 is only one instance of Surface B. Surface C is measured in two different ways: hidden-test failure, and End2End, which also includes code, proof and precondition checks. Each gap is shown on a different pipeline, language and benchmark set. The denominators also switch between conditional rates and shares of all attempts. I couldn't figure out which number measures which surface on which population, so it is unclear how to interpret the results.

[1] VERINA: Benchmarking Verifiable Code Generation. Ye et al.

[2] CLEVER: A Curated Benchmark for Formally Verified Code Generation. Thakur et al.

Questions for authors

1/ Can authors clarify what the proposed check measures beyond direct spec evaluation such as [1, 2]?

2/ Can authors clarify how the levels map to the surfaces, and which population and denominator each reported number uses?

Reviewer_vqtJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates verifier acceptance, specification binding, and specification fidelity, and proposes a Dafny check requiring postconditions to reference externally defined specification symbols. Experiments show that verifier-accepted programs can lack such references or fail hidden tests.

Critical strengths

The motivation is reasonable and the evaluation concern is relevant. The observation that most accepted Li@Opus samples lack postconditions is useful evidence that verifier acceptance needs careful interpretation.

Critical weaknesses
  1. The central metric does not establish the claimed verification gap. A complete inline postcondition fails the binding gate, while an equivalent external predicate passes. Even "ensures P(x, r) || true" satisfies the reference check without constraining the result. The authors acknowledge these limitations, but still headline a roughly fortyfold gap in verification quality. Without auditing the rejected inline specifications, this measures specification organization rather than the prevalence of vacuous proofs. The paper does not establish what the gate adds beyond detecting missing postconditions.

  2. The semantic-fidelity results are not validated. A verified program failing tests does not isolate specification inadequacy unless proof bypasses, excluded inputs, and integration failures are ruled out. No breakdown of the reported failures establishes this attribution. The VeriContest comparison has another unresolved issue: VCG produces Dafny certificates, whereas Post2Exe evaluates Verus postconditions. The adaptation and equivalence of evaluation criteria are not documented. These omissions affect the central empirical claims, including the reported SOTA result.

  3. The claimed "pass-rate tax" is unsupported. Table 7 compares different generated populations, without controlling for problem difficulty or generation trajectory. It cannot establish that adding specifications causes lower correctness. The underlying counts also need reconciliation: the LiveCodeBench groups in Table 7 total 9,448 samples, compared with 9,731 verified samples in Table 3.

In summary, the proposed metric is not validated as a measure of the claimed failure mode, and the experiments do not support several central conclusions. These require additional analysis and evaluation, not just clarification of the writing.

Questions for authors
  1. What fraction of gate-rejected postconditions are actually inadequate, rather than simply inline?
  2. How are Dafny contracts evaluated under the VeriContest protocol, and which checks account for the reduction from 1,316 to 535 samples?
Reviewer_PgQ8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The core idea of the paper is that verifier acceptance does not necessarily imply meaningful semantic correctness. The paper splits the verification "surfaces" into: verifier acceptance, syntactic specification binding and semantic specification fidelity. These surfaces are then tested on five benchmarks: HumanEval, MBPP-Rust, VeriContest LC, LeetCodeDataset and LiveCodeBench v6. The findings are that with Opus 4.7 tasked with synthesizing an implementation with a specification on Human Eval, the verifier acceptance reaches 99%, yet only less than 2% carry a postcondition that references a spec-defined predicate. The paper then argues that verification benchmarks should report at least one of the latter two surfaces (syntactic binding or semantic fidelity) alongside verifier acceptance.

Critical strengths
  1. The paper is asking a timely and important question about the meaning of the term verification.
Critical weaknesses
  1. The paper is very poorly written. Sentences such as: "Many existing evaluations report neither" and "The mismatch is a defect of the metric we use to summarize the pipeline, not of the pipeline itself" are confusing where they are used at best, and may be LLM-generated (containing common phrases such as "it is not X, it is Y" without any author review. The tables are not well-calibrated for the manuscript and the text is not easy to follow.
  2. I have issues with the gates B and C proposed. For example a postcondition such as ensures r == x * x would fail semantic specification fidelity (surface C) but is perfectly capturing the behaviour of the function. Additionally, a spefication with a post-condition ensures P(x) where P == true would pass the pipeline without ensuring any guarantess. Hence, I do not believe the surfaces B and C are a "necessary-condition heuristic" as claimed by the paper.
  3. Internal inconsistencies. VCG claims to have used 7 agents in the pipeline but only 4 are named in the manuscript. The per-group counts in Table 7 also appear to not add up to the totals in Table 3: 1,032 + 8,416 = 9,448 ≠ 9,731 (LiveCodeBench v6) and 213 + 1,995 = 2,208 ≠ 2,217 (LeetCodeDataset).
Questions for authors

Please address the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.