- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that the verifier pass rate overstates what LLM-generated code actually verifies, since verified code may have no postcondition or a wrong specification. It proposes a static Dafny check that requires a postcondition to reference a predicate in the spec file. The check is applied to a reproduction of Li et al.'s pipeline and to the authors' own pipeline.
Critical strengths
1/ The reproduction of Li et al.'s pipeline is a concrete example that verifier acceptance alone can be misleading.
Critical weaknesses
1/ The contribution is overstated. Existing work already evaluates spec quality directly. VERINA [1] checks soundness and completeness against reference specs, CLEVER [2] proves equivalence to reference specs to rule out vacuous ones. The proposed check doesn't add much on top of these. A missing postcondition is just the weakest spec, and whether a postcondition names a predicate from another file is a writing convention, not a measure of meaning. An equivalent inline postcondition fails the check, while a tautology that mentions the predicate passes it.
2/ The evaluation is very hard to follow. The paper defines three surfaces (A, B, C), but the Li et al. results are reported in three levels that don't match them. Level 2 (any postcondition) has no corresponding surface, and Level 3 is only one instance of Surface B. Surface C is measured in two different ways: hidden-test failure, and End2End, which also includes code, proof and precondition checks. Each gap is shown on a different pipeline, language and benchmark set. The denominators also switch between conditional rates and shares of all attempts. I couldn't figure out which number measures which surface on which population, so it is unclear how to interpret the results.
[1] VERINA: Benchmarking Verifiable Code Generation. Ye et al.
[2] CLEVER: A Curated Benchmark for Formally Verified Code Generation. Thakur et al.
Questions for authors
1/ Can authors clarify what the proposed check measures beyond direct spec evaluation such as [1, 2]?
2/ Can authors clarify how the levels map to the surfaces, and which population and denominator each reported number uses?