- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents CoVeri, an LLM-based framework for compositional annotation synthesis in program verification. Locally verifiable contracts may still fail to compose across interacting functions and requires coordinated revisions to dependent annotations. CoVeri combines bottom-up symbolic reasoning to identify implementation-supported properties with top-down demand analysis to trace failed verification obligations. These analyses guide LLMs to jointly revise function contracts and proof annotations while preserving executable code and fixed specifications. CoVeri is evaluated on across three LLM backbones on DafnyComp, VeriEquivBench, and a newly constructed SWE-Dafny benchmark, with additional experiments on DafnyBench. The results show consistent improvements over existing baselines, and the ablation studies demonstrate the contributions of bidirectional guidance and joint revision.
Critical strengths
-
Well-motivated problem and well-designed approach. The paper addresses an important challenge in program verification that locally verified annotations may fail to compose across interacting functions. The proposed combination of bottom-up symbolic reasoning, top-down demand analysis, and joint contract revision provides a systematic approach to resolving such problem.
-
Strong evaluation results and comprehensive ablations. CoVeri consistently outperforms the evaluated baselines across multiple benchmarks and LLM backbones, with particularly substantial improvements for smaller models. The ablation studies further demonstrate the contributions of bidirectional guidance and joint revision, providing convincing evidence that the performance gains arise from the proposed design.
Critical weaknesses
W1: Limited novelty and insufficient differentiation from prior work. The core ideas of CoVeri, including symbolic reasoning, proof-guided annotation synthesis, and compositional verification, have been explored in prior works such as AutoSpec [1], SESpec [2], Preguss [3], and ACSE [4]. The main contribution appears to be the integration of existing works rather than new mechanisms. Also, the paper does not sufficiently distinguish its approach from these closely related works, making it difficult to assess the significance of its methodological novelty.
W2: Insufficient validation of SWE-Dafny. SWE-Dafny is constructed by using an LLM to translate Python from SWE-bench Verified into Dafny and generate corresponding specifications. Although the authors manually review the translations and verify all programs, the paper provides insufficient evidence that the generated implementations and specifications faithfully preserve the original semantics. Successful verification on SWE-Dafny does not necessarily demonstrate the ability to verify the intended behavior of real-world software. This raises a substantial concern about the benchmark's construct validity and weakens the paper's claims of real-world generalizability.
[1] Wen, Cheng, et al. Enchanting program specification synthesis by large language models using static analysis and program verification. 2024.
[2] Yang, Fanpeng, et al. Integrating symbolic execution with llms for automated generation of program specifications. 2025.
[3] Wang, Zhongyi, et al. "Preguss: It analyzes, it specifies, it verifies. 2025.
[4] Lu, Weijie, et al. Array-Carrying Symbolic Execution for Function Contract Generation. 2026.
Questions for authors
Q1: Methodological novelty. Could the authors provide a more detailed discussion of how CoVeri differs from closely related works such as AutoSpec, SESpec, and Preguss? Including additional comparisons with these methods would help better establish CoVeri's contributions and empirical advantages.
Q2: Validity of SWE-Dafny. Since SWE-Dafny is released, I cannot assess its quality. How do the authors ensure that the LLM-translated Dafny implementations and specifications faithfully preserve the semantics and requirements of the original Python programs? Was any systematic validation performed beyond manual review and Dafny verification?
Q3: Quality of generated specifications. On DafnyComp with Qwen3.5-122B, only 37 of the 175 verified programs (21.1%) satisfy the Superior Specifications criterion. Could the authors explain this substantial gap and provide a breakdown of failures for the two conditions in Equation 4? This would help clarify the limitations of the generated specifications and whether relaxing generated preconditions contributes to the issue.
6 comments
Respond to Reviewer 4lay (2)
W1/Q1 (continued)
Compared with AutoSpec, AutoSpec+, and Preguss, CoVeri achieves higher verification coverage on both benchmarks, with lower average token usage and fewer requests per task.
To test whether call-graph grouping and verifier-guided joint editing can match CoVeri's verification rates, we use an additional Structure-guided control. The control fixes call-graph groups before refinement and lets the LLM infer cross-interface relationships from code and verifier feedback and jointly edit declarations within each group. Across four DafnyComp runs, Full CoVeri achieved final mean verification rates 12.8 and 10.7 percentage points higher than this control with the 35B and 122B backbones, respectively.
W2/Q2: SWE-Dafny Fidelity and Defect Sensitivity
We thank the reviewer for raising this concern. To address this concern, we supplemented manual review and Dafny verification with source-repository regression tests and representative pre-repair faulty controls.
Each task is derived from the SWE-bench instance's specified base commit and official repair patch. Its fixed entry-point requirements are constructed from the original issue requirements and repaired source, covering the selected functionality's input domain and required behavior. Dependencies outside the modeled implementation are represented by interfaces with explicit contracts. All 100 ground-truth programs passed Dafny verification. After restoring the necessary native helper interfaces, 100/100 repaired Dafny tasks passed the 318 regression test cases from the original repository. The original test assertions were unchanged, and the tests exercised the compiled Dafny implementations. This provides behavioral evidence tied to source-repository requirements.
We further constructed representative pre-repair faulty implementations based on the original repair patches. After aligning runtime semantics and supplementing the core contracts of two tasks, we established 75 eligible faulty controls. With the corresponding specifications and proof annotations held fixed during evaluation, all 75 buggy controls failed to be verified (75/75). This result measures non-verification under the fixed specifications and proof annotations. The remaining 25 tasks were excluded from this control set because their current types, interfaces, or models do not retain mechanisms needed to construct reliable corresponding faulty implementations, such as legacy APIs, missing values, exceptions, or side effects. Together, these checks provide quantitative evidence of fidelity for the tested behavior and sensitivity to representative defects.
Q3: SSR Gaps and Precondition Relaxation
We thank the reviewer for requesting this breakdown. We follow Re:Form[6] in distinguishing verification success from superiority with respect to reference specifications. SSR requires the entire program, including the entry point and all helper functions and methods, to pass Dafny verification, and every evaluated declaration to satisfy both conditions in Eq. (4): reference-input coverage, , and reference-guarantee implication, .
To check for verification bypasses, we audited the 175 verified outputs and found no newly introduced assumptions or verification-control attributes. These findings support interpreting Verified as successful contract and proof construction while preserving the program and its fixed constraints, with no evidence of these verification-bypass mechanisms.
We further audited the 138 verified programs that did not meet SSR in the original Qwen3.5-122B-A10B run. The categories below can overlap.
| Diagnostic category | Count |
|---|---|
| Restricted reference-input coverage (failure of ) | 7 programs |
| Insufficient reference guarantees (failure of ) | 137 programs |
| Inconclusive implication checks | 10 checks across 7 programs |
Among the seven programs with input-domain coverage gaps, six are affected only at auxiliary declarations and one at the main entry point. The latter is Task218: the reference precondition is 1 <= o <= 250, whereas the generated precondition is o >= 5, excluding four reference-admissible inputs. In 97/137 programs (70.8%) with insufficient reference guarantees, the confirmed guarantee gaps affect at most two declarations. The 10 inconclusive checks arise from counterexample-search timeouts, unsupported character conversions, or limited sequence-search scope in the supplementary diagnosis. All seven affected programs also have other confirmed gaps, so these inconclusive checks do not determine their program-level non-SSR classification.
Respond to Reviewer 4lay (3)
Q3 (continued)
Moreover, 115/138 programs (83.3%) passed both entry-point checks, with failed comparisons confined to auxiliary declarations in these programs. Together with whole-program verification, this shows that their existing contracts and proof information suffice to establish the stated reference entry-point properties. DafnyComp supplies implementations and retained body annotations for synthesis. SSR uses the held-out reference contracts to check whether each evaluated declaration in a verified program admits the reference input domain and implies the reference guarantee on that domain.
Precondition relaxation aims to improve reference-input coverage, and only 7 of the 138 non-SSR programs have confirmed gaps in this condition. We additionally compare CoVeri with No-relax, which disables relaxation of generated preconditions while retaining both analyses and joint edits. The following table reports Verified and SSR success counts on DafnyComp with Qwen3.5-122B-A10B. Both metrics are reported as mean sample SD across four runs (the original run and three additional runs).
| Method | Verified | SSR successes |
|---|---|---|
| CoVeri | 181.75 8.42 | 36.25 1.50 |
| No-relax | 152.25 3.20 | 29.75 2.22 |
These results support the contribution of allowing precondition relaxation to obtaining more verified programs that also satisfy both reference-specification conditions in this setting. Across four runs, allowing precondition relaxation increased the mean number of SSR successes from 29.75 to 36.25.
We hope our responses have addressed your concerns. In the revised manuscript, we have expanded the related work discussion (Section 2) and clarified CoVeri’s method and novelty (Abstract and Sections 1 and 3). We have also reported additional SWE-Dafny validation checks (Appendix B), added new experimental results (Section 4 and Appendix D), and corrected some typos. The changes are highlighted in red.
We would be happy to discuss further if you have any additional questions.
References
[1] Enchanting Program Specification Synthesis by Large Language Models Using Static Analysis and Program Verification. Cheng Wen et al. CAV, 2024.
[2] Integrating Symbolic Execution with LLMs for Automated Generation of Program Specifications. Fanpeng Yang et al. arXiv preprint, 2025.
[3] A Tale of 1001 LoC: Potential Runtime Error-Guided Specification Synthesis for Verifying Large-Scale Programs. Zhongyi Wang et al. OOPSLA, 2026.
[4] Array-Carrying Symbolic Execution for Function Contract Generation. Weijie Lu et al. FM, 2026.
[5] Formally Specifying the Intended Behavior of the Program: LLM-Driven Neuro-Symbolic Program Specification Synthesis. Cheng Wen et al. ACL System Demonstrations, 2026.
[6] Re:Form — Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny. Chuanhao Yan et al. TMLR, 2026.
Respond to Reviewer 4lay (1)
Dear Reviwer 4lay:
We sincerely thank you for your time and thoughtful review. We would like to address your concerns below.
W1/Q1: Methodological Novelty and Baseline Comparisons
We thank the reviewer for the discussion of our methodological contribution and the suggestions regarding related approaches. CoVeri's principal synthesis advance is to turn unmet use-site obligations into explicit cross-interface synthesis targets, with expressible demands formulated as temporary local proof queries. By integrating compositional symbolic reasoning with LLM synthesis, CoVeri uses feedback from these queries and bottom-up deductions about the implementation to guide the construction of caller-required contracts and the internal annotations needed to establish them. This makes cross-interface compatibility a constructive basis for specification discovery and refinement, enabling CoVeri to actively identify proof gaps and construct intermediate abstractions that support the whole-program proof.
CoVeri explicitly connects implementation-side properties to contextualized use-site demands. This connection guides which relationships cross-interface candidates should express and which contracts and internal annotations should be constructed together. Specifically, bottom-up reasoning derives implementation-side relations from code, current contracts, and invariants, turns expressible properties into interface candidates for verification, and supplies evidence for further synthesis; top-down demand analysis connects unmet obligations to interfaces that may provide the required information while retaining calling conditions, parameter mappings, and demand provenance. When a demand can be mapped into the callee's vocabulary, bounded local verification of the resulting obligations can reveal gaps in the internal proof of the required guarantee. Call-graph grouping defines which declarations can be revised together. CoVeri uses implementation-side relations, contextualized use-site demands, and available feedback from local proof queries to guide the joint construction of mutually supporting interface contracts and internal proof annotations. Updated summaries then support subsequent compositional reasoning.
AutoSpec / AutoSpec+. AutoSpec[1] generates specifications bottom-up along function and loop dependencies, validates the candidates and target assertions, and retains verified specifications for further synthesis; AutoSpec+[5] additionally repairs failing components using verification feedback. CoVeri explicitly derives implementation-side symbolic relations and contextualized use-site demands to guide what the next annotations should express, connecting the construction of interface guarantees with the internal proof annotations needed to establish them.
Preguss. Preguss[3] generates host and callee annotations in separate LLM calls using the failed assertion, relevant code, and verifier feedback, then checks the candidates. CoVeri additionally checks expressible caller demands as local proof queries in the callee before proposing replacement annotations, exposing internal proof gaps that, together with implementation relations, guide the construction of mutually supporting contracts and invariants.
SESpec. SESpec[2] uses symbolic states and templates to guide LLM invariant generation, recursively generates and reuses callee summaries, and derives function postconditions from exit states. CoVeri additionally uses caller-required guarantees to direct the construction of stronger contracts and supporting internal annotations, even when a callee already verifies under its existing contract but its summary remains insufficient for the caller.
ACSE. ACSE[4] extracts contracts from the behavior captured by symbolic execution and invariant generators, without an LLM. When current summaries are insufficient for a caller, CoVeri can use that caller's requirements to drive LLM synthesis of additional interface guarantees and the internal annotations needed to prove them, extending synthesis beyond extracting consequences of the available summaries.
To evaluate the practical value of this synthesis design, we include results from Dafny adaptations of AutoSpec, AutoSpec+, and Preguss. All rows below use Qwen3.5-35B-A3B. Verified reports the number and percentage of tasks that pass verification. Token usage and requests are averaged over all tasks.