acceptodds
Under review as a conference paper at ICLR 2027

Compositional Annotation Synthesis for Program Verification with LLMs

Abstract

Formal verification can establish that LLM-generated code satisfies its specification, but constructing proof annotations remains costly. With interacting functions, locally provable annotations may fail to compose: a callee's guarantee may omit facts its callers need, or a generated requirement may exceed what its callers establish. Verifier feedback identifies failures but does not by itself determine how dependent annotations should change together. We present CoVeri, which combines bottom-up symbolic reasoning over implementations and current summaries with top-down analysis that traces failed obligations to candidate interfaces. These analyses guide the LLM to strengthen guarantees, relax generated requirements, and jointly revise dependent contracts while preserving fixed specifications. Across three LLM backbones, CoVeri outperforms all evaluated baselines on DafnyComp, VeriEquivBench, and SWE-Dafny. On DafnyComp, it matches the strongest baseline while using a smaller backbone. Ablations support the complementary benefits of bidirectional guidance and joint revision.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Large Language Models, Program Verification, Compositional Reasoning, Formal Language, Annotation Synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4LayOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents CoVeri, an LLM-based framework for compositional annotation synthesis in program verification. Locally verifiable contracts may still fail to compose across interacting functions and requires coordinated revisions to dependent annotations. CoVeri combines bottom-up symbolic reasoning to identify implementation-supported properties with top-down demand analysis to trace failed verification obligations. These analyses guide LLMs to jointly revise function contracts and proof annotations while preserving executable code and fixed specifications. CoVeri is evaluated on across three LLM backbones on DafnyComp, VeriEquivBench, and a newly constructed SWE-Dafny benchmark, with additional experiments on DafnyBench. The results show consistent improvements over existing baselines, and the ablation studies demonstrate the contributions of bidirectional guidance and joint revision.

Critical strengths
  1. Well-motivated problem and well-designed approach. The paper addresses an important challenge in program verification that locally verified annotations may fail to compose across interacting functions. The proposed combination of bottom-up symbolic reasoning, top-down demand analysis, and joint contract revision provides a systematic approach to resolving such problem.

  2. Strong evaluation results and comprehensive ablations. CoVeri consistently outperforms the evaluated baselines across multiple benchmarks and LLM backbones, with particularly substantial improvements for smaller models. The ablation studies further demonstrate the contributions of bidirectional guidance and joint revision, providing convincing evidence that the performance gains arise from the proposed design.

Critical weaknesses

W1: Limited novelty and insufficient differentiation from prior work. The core ideas of CoVeri, including symbolic reasoning, proof-guided annotation synthesis, and compositional verification, have been explored in prior works such as AutoSpec [1], SESpec [2], Preguss [3], and ACSE [4]. The main contribution appears to be the integration of existing works rather than new mechanisms. Also, the paper does not sufficiently distinguish its approach from these closely related works, making it difficult to assess the significance of its methodological novelty.

W2: Insufficient validation of SWE-Dafny. SWE-Dafny is constructed by using an LLM to translate Python from SWE-bench Verified into Dafny and generate corresponding specifications. Although the authors manually review the translations and verify all programs, the paper provides insufficient evidence that the generated implementations and specifications faithfully preserve the original semantics. Successful verification on SWE-Dafny does not necessarily demonstrate the ability to verify the intended behavior of real-world software. This raises a substantial concern about the benchmark's construct validity and weakens the paper's claims of real-world generalizability.

[1] Wen, Cheng, et al. Enchanting program specification synthesis by large language models using static analysis and program verification. 2024.

[2] Yang, Fanpeng, et al. Integrating symbolic execution with llms for automated generation of program specifications. 2025.

[3] Wang, Zhongyi, et al. "Preguss: It analyzes, it specifies, it verifies. 2025.

[4] Lu, Weijie, et al. Array-Carrying Symbolic Execution for Function Contract Generation. 2026.

Questions for authors

Q1: Methodological novelty. Could the authors provide a more detailed discussion of how CoVeri differs from closely related works such as AutoSpec, SESpec, and Preguss? Including additional comparisons with these methods would help better establish CoVeri's contributions and empirical advantages.

Q2: Validity of SWE-Dafny. Since SWE-Dafny is released, I cannot assess its quality. How do the authors ensure that the LLM-translated Dafny implementations and specifications faithfully preserve the semantics and requirements of the original Python programs? Was any systematic validation performed beyond manual review and Dafny verification?

Q3: Quality of generated specifications. On DafnyComp with Qwen3.5-122B, only 37 of the 175 verified programs (21.1%) satisfy the Superior Specifications criterion. Could the authors explain this substantial gap and provide a breakdown of failures for the two conditions in Equation 4? This would help clarify the limitations of the generated specifications and whether relaxing generated preconditions contributes to the issue.

6 comments
AuthorsOpenReview ↗

Respond to Reviewer 4lay (1)

Dear Reviwer 4lay:

We sincerely thank you for your time and thoughtful review. We would like to address your concerns below.

W1/Q1: Methodological Novelty and Baseline Comparisons

We thank the reviewer for the discussion of our methodological contribution and the suggestions regarding related approaches. CoVeri's principal synthesis advance is to turn unmet use-site obligations into explicit cross-interface synthesis targets, with expressible demands formulated as temporary local proof queries. By integrating compositional symbolic reasoning with LLM synthesis, CoVeri uses feedback from these queries and bottom-up deductions about the implementation to guide the construction of caller-required contracts and the internal annotations needed to establish them. This makes cross-interface compatibility a constructive basis for specification discovery and refinement, enabling CoVeri to actively identify proof gaps and construct intermediate abstractions that support the whole-program proof.

CoVeri explicitly connects implementation-side properties to contextualized use-site demands. This connection guides which relationships cross-interface candidates should express and which contracts and internal annotations should be constructed together. Specifically, bottom-up reasoning derives implementation-side relations from code, current contracts, and invariants, turns expressible properties into interface candidates for verification, and supplies evidence for further synthesis; top-down demand analysis connects unmet obligations to interfaces that may provide the required information while retaining calling conditions, parameter mappings, and demand provenance. When a demand can be mapped into the callee's vocabulary, bounded local verification of the resulting obligations can reveal gaps in the internal proof of the required guarantee. Call-graph grouping defines which declarations can be revised together. CoVeri uses implementation-side relations, contextualized use-site demands, and available feedback from local proof queries to guide the joint construction of mutually supporting interface contracts and internal proof annotations. Updated summaries then support subsequent compositional reasoning.

AutoSpec / AutoSpec+. AutoSpec[1] generates specifications bottom-up along function and loop dependencies, validates the candidates and target assertions, and retains verified specifications for further synthesis; AutoSpec+[5] additionally repairs failing components using verification feedback. CoVeri explicitly derives implementation-side symbolic relations and contextualized use-site demands to guide what the next annotations should express, connecting the construction of interface guarantees with the internal proof annotations needed to establish them.

Preguss. Preguss[3] generates host and callee annotations in separate LLM calls using the failed assertion, relevant code, and verifier feedback, then checks the candidates. CoVeri additionally checks expressible caller demands as local proof queries in the callee before proposing replacement annotations, exposing internal proof gaps that, together with implementation relations, guide the construction of mutually supporting contracts and invariants.

SESpec. SESpec[2] uses symbolic states and templates to guide LLM invariant generation, recursively generates and reuses callee summaries, and derives function postconditions from exit states. CoVeri additionally uses caller-required guarantees to direct the construction of stronger contracts and supporting internal annotations, even when a callee already verifies under its existing contract but its summary remains insufficient for the caller.

ACSE. ACSE[4] extracts contracts from the behavior captured by symbolic execution and invariant generators, without an LLM. When current summaries are insufficient for a caller, CoVeri can use that caller's requirements to drive LLM synthesis of additional interface guarantees and the internal annotations needed to prove them, extending synthesis beyond extracting consequences of the available summaries.

To evaluate the practical value of this synthesis design, we include results from Dafny adaptations of AutoSpec, AutoSpec+, and Preguss. All rows below use Qwen3.5-35B-A3B. Verified reports the number and percentage of tasks that pass verification. Token usage and requests are averaged over all tasks.

BenchmarkMethodVerifiedTotal tokens/task (k)Requests/task
DafnyComp (N=300)CoVeri164 (54.7%)106.55.92
AutoSpec81 (27.0%)446.831.75
AutoSpec+97 (32.3%)325.523.52
Preguss96 (32.0%)259.028.38
SWE-Dafny (N=100)CoVeri70 (70.0%)66.63.95
AutoSpec59 (59.0%)168.111.74
AutoSpec+62 (62.0%)109.28.67
Preguss52 (52.0%)126.212.78
AuthorsOpenReview ↗

Respond to Reviewer 4lay (2)

W1/Q1 (continued)

Compared with AutoSpec, AutoSpec+, and Preguss, CoVeri achieves higher verification coverage on both benchmarks, with lower average token usage and fewer requests per task.

To test whether call-graph grouping and verifier-guided joint editing can match CoVeri's verification rates, we use an additional Structure-guided control. The control fixes call-graph groups before refinement and lets the LLM infer cross-interface relationships from code and verifier feedback and jointly edit declarations within each group. Across four DafnyComp runs, Full CoVeri achieved final mean verification rates 12.8 and 10.7 percentage points higher than this control with the 35B and 122B backbones, respectively.

W2/Q2: SWE-Dafny Fidelity and Defect Sensitivity

We thank the reviewer for raising this concern. To address this concern, we supplemented manual review and Dafny verification with source-repository regression tests and representative pre-repair faulty controls.

Each task is derived from the SWE-bench instance's specified base commit and official repair patch. Its fixed entry-point requirements are constructed from the original issue requirements and repaired source, covering the selected functionality's input domain and required behavior. Dependencies outside the modeled implementation are represented by interfaces with explicit contracts. All 100 ground-truth programs passed Dafny verification. After restoring the necessary native helper interfaces, 100/100 repaired Dafny tasks passed the 318 regression test cases from the original repository. The original test assertions were unchanged, and the tests exercised the compiled Dafny implementations. This provides behavioral evidence tied to source-repository requirements.

We further constructed representative pre-repair faulty implementations based on the original repair patches. After aligning runtime semantics and supplementing the core contracts of two tasks, we established 75 eligible faulty controls. With the corresponding specifications and proof annotations held fixed during evaluation, all 75 buggy controls failed to be verified (75/75). This result measures non-verification under the fixed specifications and proof annotations. The remaining 25 tasks were excluded from this control set because their current types, interfaces, or models do not retain mechanisms needed to construct reliable corresponding faulty implementations, such as legacy APIs, missing values, exceptions, or side effects. Together, these checks provide quantitative evidence of fidelity for the tested behavior and sensitivity to representative defects.

Q3: SSR Gaps and Precondition Relaxation

We thank the reviewer for requesting this breakdown. We follow Re:Form[6] in distinguishing verification success from superiority with respect to reference specifications. SSR requires the entire program, including the entry point and all helper functions and methods, to pass Dafny verification, and every evaluated declaration to satisfy both conditions in Eq. (4): reference-input coverage, Pref⇒PcP_{\mathrm{ref}}\Rightarrow P_c, and reference-guarantee implication, Pref∧Qc⇒QrefP_{\mathrm{ref}}\land Q_c\Rightarrow Q_{\mathrm{ref}}.

To check for verification bypasses, we audited the 175 verified outputs and found no newly introduced assumptions or verification-control attributes. These findings support interpreting Verified as successful contract and proof construction while preserving the program and its fixed constraints, with no evidence of these verification-bypass mechanisms.

We further audited the 138 verified programs that did not meet SSR in the original Qwen3.5-122B-A10B run. The categories below can overlap.

Diagnostic categoryCount
Restricted reference-input coverage (failure of Pref⇒PcP_{\mathrm{ref}}\Rightarrow P_c)7 programs
Insufficient reference guarantees (failure of Pref∧Qc⇒QrefP_{\mathrm{ref}}\land Q_c\Rightarrow Q_{\mathrm{ref}})137 programs
Inconclusive implication checks10 checks across 7 programs

Among the seven programs with input-domain coverage gaps, six are affected only at auxiliary declarations and one at the main entry point. The latter is Task218: the reference precondition is 1 <= o <= 250, whereas the generated precondition is o >= 5, excluding four reference-admissible inputs. In 97/137 programs (70.8%) with insufficient reference guarantees, the confirmed guarantee gaps affect at most two declarations. The 10 inconclusive checks arise from counterexample-search timeouts, unsupported character conversions, or limited sequence-search scope in the supplementary diagnosis. All seven affected programs also have other confirmed gaps, so these inconclusive checks do not determine their program-level non-SSR classification.

AuthorsOpenReview ↗

Respond to Reviewer 4lay (3)

Q3 (continued)

Moreover, 115/138 programs (83.3%) passed both entry-point checks, with failed comparisons confined to auxiliary declarations in these programs. Together with whole-program verification, this shows that their existing contracts and proof information suffice to establish the stated reference entry-point properties. DafnyComp supplies implementations and retained body annotations for synthesis. SSR uses the held-out reference contracts to check whether each evaluated declaration in a verified program admits the reference input domain and implies the reference guarantee on that domain.

Precondition relaxation aims to improve reference-input coverage, and only 7 of the 138 non-SSR programs have confirmed gaps in this condition. We additionally compare CoVeri with No-relax, which disables relaxation of generated preconditions while retaining both analyses and joint edits. The following table reports Verified and SSR success counts on DafnyComp with Qwen3.5-122B-A10B. Both metrics are reported as mean ±\pm sample SD across four runs (the original run and three additional runs).

MethodVerifiedSSR successes
CoVeri181.75 ±\pm 8.4236.25 ±\pm 1.50
No-relax152.25 ±\pm 3.2029.75 ±\pm 2.22

These results support the contribution of allowing precondition relaxation to obtaining more verified programs that also satisfy both reference-specification conditions in this setting. Across four runs, allowing precondition relaxation increased the mean number of SSR successes from 29.75 to 36.25.

We hope our responses have addressed your concerns. In the revised manuscript, we have expanded the related work discussion (Section 2) and clarified CoVeri’s method and novelty (Abstract and Sections 1 and 3). We have also reported additional SWE-Dafny validation checks (Appendix B), added new experimental results (Section 4 and Appendix D), and corrected some typos. The changes are highlighted in red.

We would be happy to discuss further if you have any additional questions.

References

[1] Enchanting Program Specification Synthesis by Large Language Models Using Static Analysis and Program Verification. Cheng Wen et al. CAV, 2024.

[2] Integrating Symbolic Execution with LLMs for Automated Generation of Program Specifications. Fanpeng Yang et al. arXiv preprint, 2025.

[3] A Tale of 1001 LoC: Potential Runtime Error-Guided Specification Synthesis for Verifying Large-Scale Programs. Zhongyi Wang et al. OOPSLA, 2026.

[4] Array-Carrying Symbolic Execution for Function Contract Generation. Weijie Lu et al. FM, 2026.

[5] Formally Specifying the Intended Behavior of the Program: LLM-Driven Neuro-Symbolic Program Specification Synthesis. Cheng Wen et al. ACL System Demonstrations, 2026.

[6] Re:Form — Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny. Chuanhao Yan et al. TMLR, 2026.

Reviewer_i73ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents CoVeri, a method to synthesize contracts and proof annotations with Dafny. It combines bottom-up symbolic reasoning with top-down analysis of failed proof obligations to identify related interfaces and guide their joint repair. Generated guarantees can be strengthened and generated requirements relaxed, while fixed specifications remain protected. The evaluation covers contract generation and proof completion under fixed specifications across three LLM backbones. The paper also introduces SWE-Dafny, comprising 100 verification tasks modeled from repaired SWE-bench programs.

Critical strengths

The paper addresses a concrete problem in annotation synthesis: individually provable contracts may not provide what their callers need. The motivating example clearly shows why strengthening a producer’s guarantee or relaxing a consumer’s generated requirement alone may be insufficient. CoVeri gives the model implementation facts and caller demands to guide these changes. The evaluation against ablated versions supports the usefulness of this guidance.

Critical weaknesses
  1. The technical novelty over existing compositional specification synthesis remains unclear. AutoSpec already generates annotations bottom-up and checks whether they support the desired properties. Preguss performs obligation-guided interprocedural specification synthesis. Both are cited, but the discussion does not sufficiently explain what their mechanisms cannot accomplish that CoVeri enables. ConVer [1], which is not discussed, also uses LLM-generated contracts and alternates system-level sufficiency checks with implementation-level contract checks. The broad ideas of composition and implementation/use-side consistency are established.
  2. The contract-generation results leave the quality of many successful outputs unresolved. On DafnyComp with Qwen3.5-122B-A10B, 175 programs verify, but only 37 also meet the Superior Specifications criterion. CoVeri improves this stronger metric over the baselines, which is encouraging. However, the paper does not explain whether the remaining cases have overly restrictive preconditions, incomplete postconditions, or reference checks that the prover cannot discharge. This is not saying those outputs are necessarily vacuous, but it limits what can be concluded from the table numbers.
  3. The evaluation does not include comparison with frontier models and general-purpose coding agents. All experiments use Qwen3.5 or ReForm backbones, and the agentic baselines are task-specific adaptations rather than native coding-agent workflows. A stronger agent with direct access to Dafny and debugging info may be able to identify dependencies and repair the proofs.
Questions for authors
  1. Have you evaluated a general coding agent such as Codex or Claude Code with direct access to Dafny? Under comparable budgets and the same editing restrictions, does CoVeri still improve success rate or cost over ordinary agent-driven refinement?
  2. How often does a successful repair require both strengthening a producer’s guarantee and relaxing a consumer’s generated requirement? Does disabling precondition relaxation, while retaining joint edits and both analyses, substantially reduce success?
  3. On DafnyComp, 175 programs verify with Qwen3.5-122B-A10B, but only 37 satisfy Superior Specifications. Among the remaining cases, how many contain restrictive preconditions, insufficient postconditions, or inconclusive reference checks? Does CoVeri have a systematic method to reduce and mitigate reward hacking, e.g., like the lynette tool in AutoVerus[1]?

[1] Yang, Chenyuan, Xuheng Li, Md Rakib Hossain Misu, Jianan Yao, Weidong Cui, Yeyun Gong, Chris Hawblitzel et al. "Autoverus: Automated proof generation for rust code." Proceedings of the ACM on Programming Languages 9, no. OOPSLA2 (2025): 3454-3482.

7 comments
AuthorsOpenReview ↗

Respond to Reviewer i73Z (1)

Dear Reviwer i73Z:

We sincerely thank you for your time and thoughtful review. We would like to address your concerns below.

W1: Novelty and Baseline Comparisons

We thank the reviewer for the discussion of our methodological contribution and the suggestions regarding related approaches. CoVeri's principal synthesis advance is to turn unmet use-site obligations into explicit cross-interface synthesis targets, with expressible demands formulated as temporary local proof queries. By integrating compositional symbolic reasoning with LLM synthesis, CoVeri uses feedback from these queries and bottom-up deductions about the implementation to guide the construction of caller-required contracts and the internal annotations needed to establish them. This makes cross-interface compatibility a constructive basis for specification discovery and refinement, enabling CoVeri to actively identify proof gaps and construct intermediate abstractions that support the whole-program proof.

CoVeri explicitly connects implementation-side properties to contextualized use-site demands. This connection guides which relationships cross-interface candidates should express and which contracts and internal annotations should be constructed together. Specifically, bottom-up reasoning derives implementation-side relations from code, current contracts, and invariants, turns expressible properties into interface candidates for verification, and supplies evidence for further synthesis; top-down demand analysis connects unmet obligations to interfaces that may provide the required information while retaining calling conditions, parameter mappings, and demand provenance.

When a demand can be mapped into the callee's vocabulary, bounded local verification of the resulting obligations can reveal gaps in the internal proof of the required guarantee. Call-graph grouping defines which declarations can be revised together. CoVeri uses implementation-side relations, contextualized use-site demands, and available feedback from local proof queries to guide the joint construction of mutually supporting interface contracts and internal proof annotations. Updated summaries then support subsequent compositional reasoning.

AutoSpec[1] generates specifications bottom-up along function and loop dependencies, validates the candidates and target assertions, and retains verified specifications for further synthesis; AutoSpec+[2] additionally repairs failing components using verification feedback. CoVeri explicitly derives implementation-side symbolic relations and contextualized use-site demands to guide what the next annotations should express, connecting the construction of interface guarantees with the internal proof annotations needed to establish them.

ConVer[3] uses an LLM to derive contracts from system-level properties and alternates system-level and implementation-level checks, using counterexamples and accumulated ICE information to guide contract refinement. CoVeri's distinctive step is to formulate temporary local proof queries from explicitly derived, contextualized use-site demands. Even when a callee verifies under its current contract, these queries can expose missing internal proof support for the caller-required guarantee, giving the next LLM-guided revision an explicit interface target and corresponding proof feedback.

Preguss[4] generates host and callee annotations in separate LLM calls using the failed assertion, relevant code, and verifier feedback, then checks the candidates. CoVeri additionally checks expressible caller demands as temporary local proof queries in the callee before the next LLM-guided annotation revision; the resulting feedback can expose internal proof gaps and, together with implementation-side relations, guide the construction of mutually supporting contracts and invariants.

To evaluate the practical value of this synthesis design, we include results from Dafny adaptations of AutoSpec, AutoSpec+, ConVer, and Preguss. All rows below use Qwen3.5-35B-A3B. Verified reports the number and percentage of tasks that pass verification. Token usage and requests are averaged over all tasks.

BenchmarkMethodVerifiedTotal tokens/task (k)Requests/task
DafnyComp (N=300)CoVeri164 (54.7%)106.55.92
AutoSpec81 (27.0%)446.831.75
AutoSpec+97 (32.3%)325.523.52
Preguss96 (32.0%)259.028.38
SWE-Dafny (N=100)CoVeri70 (70.0%)66.63.95
AutoSpec59 (59.0%)168.111.74
AutoSpec+62 (62.0%)109.28.67
ConVer49 (49.0%)25.53.66
Preguss52 (52.0%)126.212.78

Compared with AutoSpec, AutoSpec+, and Preguss, CoVeri achieves higher verification coverage on both benchmarks, with lower average token usage and fewer requests per task. On SWE-Dafny, CoVeri verifies 70/100 tasks compared with ConVer's 49/100, improving verification coverage by 21 percentage points.

AuthorsOpenReview ↗

Respond to Reviewer i73Z (2)

W1 (continued)

To test whether call-graph grouping and verifier-guided joint editing can match CoVeri's verification rates, we use an additional Structure-guided control. The control fixes call-graph groups before refinement and lets the LLM infer cross-interface relationships from code and verifier feedback and jointly edit declarations within each group. Across four DafnyComp runs, Full CoVeri achieved final mean verification rates 12.8 and 10.7 percentage points higher than this control with the 35B and 122B backbones, respectively.

W2/Q3: Specification Quality and Reward Hacking

We thank the reviewer for the questions about specification quality and reward hacking, and address W2 and Q3 together. To clarify the implications of the 37/175 SSR result, we analyze three aspects of the original 122B run cited by the reviewer: verification integrity, the locations of specification gaps, and improvements over the baseline.

First, CoVeri applies three safeguards to mitigate reward hacking:

  • Preserving executable code: Parser-based checks preserve executable code, function signatures, and protected definitions, preventing the model from simplifying the implementation merely to make verification succeed.
  • Protecting fixed specifications and input domains: Candidates cannot exclude a task's protected entry domain through conditions such as requires false, or replace an existing fixed guarantee with ensures true. New annotations must respect declaration-specific editing permissions.
  • Rejecting proof bypasses: We reject introduced assume and axiom constructs, prohibited attribute changes, and invalid uses of decreases *, preventing candidates from assuming conclusions or bypassing required termination proofs.

The audit of the 175 outputs found no newly introduced assumptions or verification-control attributes. These findings support interpreting Verified as successful contract and proof construction while preserving the program and its fixed constraints, with no evidence of these verification-bypass mechanisms. These safeguards enforce task-fixed requirements and editing permissions; the adequacy of newly generated contracts relative to the reference specifications is assessed separately through SSR.

Second, we follow Re:Form[5] in distinguishing verification success from superiority with respect to reference specifications. SSR requires the entire program, including the entry point and all helper functions and methods, to pass Dafny verification. It also requires every evaluated declaration to pass the reference-specification checks. DafnyComp supplies implementations and retained body annotations for synthesis. SSR uses the held-out reference contracts to check whether each evaluated declaration in a verified program admits the reference input domain and implies the reference guarantee on that domain.

We audited the 138 verified programs that did not meet SSR in the original Qwen3.5-122B-A10B run. The categories below can overlap.

Diagnostic categoryCount
Restricted reference-input coverage7 programs
Insufficient reference guarantees137 programs
Inconclusive implication checks10 checks across 7 programs

The domain-coverage and guarantee-gap categories overlap in six programs; all 138 programs have at least one confirmed gap.

Among the 7 programs with input-domain coverage gaps, 6 are affected only at auxiliary declarations and 1 at the main entry point. The only program with a main-entry input-coverage gap is Task218: the reference precondition is 1 <= o <= 250, whereas the generated precondition is o >= 5, excluding four reference-admissible inputs. In 97/137 (70.8%) of these programs with insufficient reference guarantees, the confirmed guarantee gaps affect at most two declarations. The 10 inconclusive checks arise from counterexample-search timeouts, unsupported character conversions, or limited sequence-search scope in the supplementary diagnosis; all 7 affected programs also have other confirmed gaps, so these inconclusive checks do not affect their program-level non-SSR classification.

Moreover, we found that 115/138 programs (83.3%) passed both entry-point checks: coverage of the reference entry-point input domain and implication of its stated output guarantees on that domain. In these 115 programs, the failed comparisons were confined to auxiliary declarations. Together with whole-program verification, this shows that the existing contracts and proof information in these 115 programs suffice to establish the stated entry-point properties.

AuthorsOpenReview ↗

Respond to Reviewer i73Z (3)

W2/Q3 (continued)

Finally, CoVeri's relative gain is larger for SSR successes than for Verified counts.

Qwen3.5-122B-A10B, DafnyCompVerifiedSSR successesSSR / Verified
AxDafny1642213.4%
CoVeri1753721.1%

Compared with AxDafny, CoVeri increases the Verified count by 6.7% and the SSR success count by 68.2%, while raising the proportion of verified outputs satisfying SSR from 13.4% to 21.1%. These results support improvements in both the number of verified programs and the proportion of verified outputs meeting the reference-specification criterion.

W3/Q1: Frontier Models and Codex

BackboneMethodVerifiedSSRTokens/task
GPT-5.6 LunaBase83/300 (27.67%)105.28k
Base Refine214/300 (71.33%)2021.24k
DafnyPro227/300 (75.67%)1119.60k
AxDafny231/300 (77.00%)1728.50k
Codex237/300 (79.00%)12107.54k
CoVeri253/300 (84.33%)2615.68k
GPT-5.6 TerraBase112/300 (37.33%)115.16k
Base Refine235/300 (78.33%)1315.62k
DafnyPro244/300 (81.33%)414.81k
AxDafny241/300 (80.33%)624.39k
Codex246/300 (82.00%)9106.97k
CoVeri269/300 (89.67%)2815.49k

We thank the reviewer for highlighting the need to evaluate stronger backbones and general-purpose coding agents. We have added experiments using GPT-5.6 Luna and GPT-5.6 Terra on all 300 DafnyComp tasks, comparing CoVeri with Base, Base Refine, DafnyPro, AxDafny, and Codex. We use Codex in its native workflow as a general-purpose coding agent, providing the original task and annotation-editing rules in an isolated sandbox. Codex autonomously edits files, runs Dafny, and repairs failures based on verifier diagnostics. To align the round budget with CoVeri, we explicitly instruct Codex in the prompt to perform one initialization round followed by four refinement rounds.

CoVeri achieves the highest Verified and SSR counts among all evaluated methods with both backbones. Compared with Codex, CoVeri improves verification and SSR rates on both backbones. It is also worth noting that compared with Codex, these gains are accompanied by 85.42% and 85.52% lower recorded token consumption, respectively. These results provide additional evidence that CoVeri's benefits extend to the new frontier backbones and persist when compared with a general-purpose coding agent that can directly invoke Dafny and iteratively repair annotations.

Q2: Joint Contract Edits and Precondition Relaxation

We thank the reviewer for this suggestion. We report both the frequency of the two modifications and their necessity within a successful patch. Let S be the number of successful repairs on DafnyComp in the original run for each backbone. R/S is the fraction whose initial-to-final patch both strengthens a producer's guarantee and relaxes a consumer's generated requirement on the same dependency. J/S is the fraction for which reverting either modification separately, while holding all other edits fixed, causes verification failure under the fixed verifier and budget.

BackboneSR/SJ/S
Qwen3.5-35B-A3B9611/96 (11.46%)9/96 (9.38%)
Qwen3.5-122B-A10B10010/100 (10.00%)8/100 (8.00%)

Thus, in 9.38% and 8.00% of the respective successful-repair cohorts, both modifications are needed for the final patch to verify under this test.

To illustrate the reversion check, consider Qwen3.5-122B-A10B/Task 37. The producer smallestFactorization_625 strengthens its guarantee from true to result >= 0, while the consumer monotoneIncreasingDigits_738 relaxes its generated requirement from n > 0 to n >= 0. The complete patch verifies. Holding all other edits fixed, reverting only the producer change, only the consumer change, or both causes verification failure at the same call to the consumer.

Reviewer_sd9hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the mismatch between locally provable annotations and the requirements of their callers, this paper proposes CoVeri, a compositional annotation-synthesis framework for Dafny. The method combines bottom-up symbolic reasoning with top-down analysis of failed proof obligations, and groups dependent interfaces for joint revision by an LLM. Executable implementations and task-fixed specifications remain protected, while permitted internal contracts and proof annotations can be revised. Experiments cover contract generation and proof completion across three LLM backbones, including a new collection of 100 SWE-Dafny tasks. The results show improved verification rates, with ablations examining the contributions of symbolic guidance, demand analysis, and joint revision.

Critical strengths
  • S1. The paper addresses a real difficulty in modular verification: locally provable annotations may not provide the facts needed by their callers. The motivating example clearly explains why producer guarantees and generated consumer requirements sometimes need coordinated changes. Combining implementation-side symbolic context with use-side demands is a reasonable engineering design. Protection of executable code and fixed requirements, followed by whole-program verification, also provides appropriate safeguards for the generated annotations.

  • S2. The evaluation reports consistent improvements across the backbone–task combinations in Table 1 and includes several useful controls. With Qwen3.5-35B-A3B, CoVeri verifies 164 of 300 DafnyComp programs versus 112 for the strongest baseline. The Single-edit, w/o TD, w/o BU, and shared-initialization variants help assess the role of the components. Figure 5 also provides actual cost evidence: in this setting, CoVeri at two iterations exceeds DafnyPro at five while using 26.4% fewer tokens and 16.8% less time. The fixed-specification experiments and Superior Specifications metric add value beyond verification counts alone.

Critical weaknesses
  • W1. The methodological contribution appears limited relative to existing structure-guided specification synthesis and proof repair. [AutoSpec (CAV 2024)] already combines program decomposition, bottom-up specification generation, and verification of specifications and target properties, while [Laurel (OOPSLA 2025)] uses verifier diagnostics to localize missing proof information. CoVeri's bottom-up analysis uses standard symbolic relation composition and projection; its top-down analysis traces dependencies from failed obligations; and Algorithm 2 groups selected declarations by call-graph connectivity before LLM-guided repair. The main addition is how these analyses organize joint changes to producer and consumer contracts. This is a sensible extension, but the paper does not sufficiently establish a substantial synthesis advance beyond integrating these established techniques. The feasibility and use-side sufficiency conditions also express familiar modular-verification requirements. The ablations demonstrate the utility of the components more clearly than the significance of the methodological increment.

  • W2. The empirical and benchmark contributions do not yet compensate for the limited methodological novelty. The smaller-backbone gains and cost results are useful, but gains over the strongest baseline with the larger backbone are more modest: 175 versus 164 on DafnyComp, 73 versus 69 on SWE-Dafny, and 606 versus 586 on DafnyBench. The central ablations match iterations rather than actual request or token budgets, and repeated-run uncertainty is not reported. Moreover, only 37 of the 175 verified DafnyComp outputs meet the Superior Specifications criterion. This improves on the baseline and does not imply that the other outputs are incorrect, but it limits what verification counts establish about specification quality. DafnyComp has already studied the compositional task itself. SWE-Dafny adds a useful constructed dataset, with source-correspondence review and verified references, but its semantic fidelity and ability to distinguish original bugs from repairs are not quantitatively established. Overall, the evidence supports a useful implementation more strongly than a sufficiently substantial new verification capability.

Questions for authors
  • Q1. What is the principal synthesis advance over existing structure-guided specification generation and verifier-guided repair, beyond combining their analyses and allowing joint edits? Can the authors identify representative fixed-specification tasks where the proposed demand-guided coordination provides a capability that a comparable structure-aware repair strategy does not?

  • Q2. Do the benefits of Full CoVeri over Single-edit, w/o TD, and the shared-initialization refinement control persist under matched token or time budgets? Please also quantify run-to-run uncertainty, particularly for the smaller margins with the larger backbone.

  • Q3. For verified DafnyComp outputs that fail the Superior Specifications check, how often is the cause a restricted input domain, an insufficient guarantee, or an inconclusive implication check? For SWE-Dafny, can the authors quantify whether the translated fixed specifications preserve the relevant source behavior and reject representative pre-repair buggy implementations?

6 comments
AuthorsOpenReview ↗

Respond to Reviewer sd9h (1)

Dear Reviwer sd9h:

We sincerely thank you for your time and thoughtful review. We would like to address your concerns below.

W1/Q1: Method novelty and Additional baselines

We thank the reviewer for the discussion of our methodological contribution and the suggestions regarding related approaches. CoVeri's principal synthesis advance is to turn unmet use-site obligations into explicit cross-interface synthesis targets, with expressible demands formulated as temporary local proof queries. By integrating compositional symbolic reasoning with LLM synthesis, CoVeri uses feedback from these queries and bottom-up deductions about the implementation to guide the construction of caller-required contracts and the internal annotations needed to establish them. This makes cross-interface compatibility a constructive basis for specification discovery and refinement, enabling CoVeri to actively identify proof gaps and construct intermediate abstractions that support the whole-program proof.

CoVeri explicitly connects implementation-side properties to contextualized use-site demands. This connection guides which relationships cross-interface candidates should express and which contracts and internal annotations should be constructed together. Specifically, bottom-up reasoning derives implementation-side relations from code, current contracts, and invariants, turns expressible properties into interface candidates for verification, and supplies evidence for further synthesis; top-down demand analysis connects unmet obligations to interfaces that may provide the required information while retaining calling conditions, parameter mappings, and demand provenance. When a demand can be mapped into the callee's vocabulary, bounded local verification of the resulting obligations can reveal gaps in the internal proof of the required guarantee. Call-graph grouping in Algorithm 2 defines which declarations can be revised together. CoVeri uses implementation-side relations, contextualized use-site demands, and available feedback from local proof queries to guide the joint construction of mutually supporting interface contracts and internal proof annotations.

AutoSpec and AutoSpec+. AutoSpec[1] generates specifications bottom-up along function and loop dependencies, using local code and callee specifications as context. It validates candidate specifications, checks the target assertions, and retains verified specifications for further synthesis. AutoSpec+[2] additionally repairs failing components using verification feedback. CoVeri turns unmet use-site obligations into explicit targets for synthesizing the required interface guarantees and their supporting proof annotations.

Laurel. Laurel[3] targets proof completion under existing specifications. It uses Dafny diagnostics and a control-flow graph to locate sites for auxiliary assertions. CoVeri targets specification synthesis across functions, constructing contracts and invariants that jointly support the whole-program proof. Moreover, CoVeri formulates a use-site demand as a temporary local proof query to identify missing proof support for the required interface guarantee.

To evaluate the practical value of this synthesis design, we include results from Dafny adaptations of AutoSpec and AutoSpec+; all rows below use Qwen3.5-35B-A3B, Verified and SSR report counts, and costs are averaged over all tasks.

BenchmarkMethodVerifiedSSRTotal tokens/task (k)Requests/task
DafnyComp (N=300)CoVeri164 (54.7%)32106.55.92
AutoSpec81 (27.0%)18446.831.75
AutoSpec+97 (32.3%)21325.523.52
SWE-Dafny (N=100)CoVeri70 (70.0%)-66.63.95
AutoSpec59 (59.0%)-168.111.74
AutoSpec+62 (62.0%)-109.28.67

CoVeri achieves higher verification coverage on both benchmarks and more outputs meeting Superior Specifications on DafnyComp, with lower total token and request costs.

AuthorsOpenReview ↗

Respond to Reviewer sd9h (2)

W1/Q1 (continued)

To give a concrete example for Q1, we use SWE-Dafny 002 (sympy__sympy-19346) to show how semantic relationships required by the whole-program proof directly guide the discovery of intermediate contracts and invariants. In this task, the original entry contracts of Srepr, PrintSet, and PrintDict remain fixed, all requiring s == Repr(v,p), while the input domains, logical definitions, executable bodies, and termination conditions are also preserved. Initially, the summaries of the helper method Reprify describe indices and token counts but do not express token contents, leaving them insufficient to support the representation equality required by the caller. The demand record connects the fixed goal to the Reprify call, guiding the model to propose the content contract ensures s == Join(Reprs(args,p), sep, |args|); after this candidate is submitted, the verifier reports that the new postcondition remains unproved. Using the candidate DoPrint contract that relates each returned token to Repr(args[i],p), together with the loop's append operation, the model further introduces invariant tokens == Reprs(args[0..i], p) and corresponding exit assertions. The key point is that the content relationship required by the caller is carried through the helper's interface contract and internal loop invariant, allowing these generated annotations to jointly support the same fixed goal. The dictionary branch gains analogous token-content relationships, and verification progresses from 15 verified, 5 errors through 18 verified, 2 errors and 19 verified, 1 error to 20 verified, 0 errors, with fixed-specification and executable-body preservation checks passing. On the same original task with the same Qwen3.5-35B-A3B backbone, neither AutoSpec nor AutoSpec+ completed verification.

This case illustrates how compositional demands participate in candidate construction; we further use internal controls to examine whether the overall gains can be explained solely by initialization, structural organization, or a broader editing scope.

To test whether call-graph ordering and grouping or joint editing explain CoVeri's gains, we added a control that fixes the groups before refinement and allows the LLM to revise functions within each group jointly. The LLM uses the code and verifier feedback to infer what must hold across interfaces. Across four DafnyComp runs, Full CoVeri achieved final mean verification rates 12.8 and 10.7 percentage points higher than this control with the 35B and 122B backbones, respectively. This comparison supports the value of using explicit implementation relations and use-site demands to guide specification discovery when the LLM can already edit structurally related functions together.

BackboneStructure-guidedFull CoVeri
35B40.25% ±\pm 1.03 pp53.00% ±\pm 2.33 pp
122B49.92% ±\pm 1.83 pp60.58% ±\pm 2.81 pp

We further assess the roles of reasoning and coordinated revision using three additional ablation controls. CoVeri-init + Base Refine shares CoVeri's initial annotations and permits multi-declaration revision using raw verifier feedback, yet underperforms Full with both backbones, supporting the value of continued reasoning-guided refinement beyond the common initialization. w/o TD retains BU, raw Dafny diagnostics, call-graph grouping, and multi-declaration revision, yet Full achieves final mean verification rates higher by 6.0 and 8.7 percentage points across four DafnyComp runs with the 35B and 122B models, respectively, showing additional gains from use-side demand analysis beyond these retained capabilities. Single-edit retains BU and TD but permits changes to only one declaration per proposal; its lower performance than Full with both models supports using the analysis results to coordinate the construction of related annotations.

Together, these controls and the fixed-specification case support our central contribution: compositional compatibility informs not only whether candidates succeed, but also how intermediate specifications are discovered, enabling synthesis to construct mutually supporting interface contracts and internal proof annotations.

AuthorsOpenReview ↗

Respond to Reviewer sd9h (3)

W2/Q2: Run-to-Run Variability, Budget Comparisons, and SSR

We thank the reviewer for the comments on gains with the larger backbone, budget comparability, and specification quality, and address W2 and Q2 together.

First, across the original run and three additional runs with the 122B backbone, CoVeri achieves positive mean gains on all four primary benchmarks in Table 1 of the main text. The following table reports mean verified-program counts ±\pm sample SD across all four runs.

ModelMethodDafnyComp (N = 300)SpecGen (N = 186)SWE-Dafny (N = 100)SpecRefine (N = 186)
Qwen3.5-122B-A10BBase8.75 ±\pm 2.99 (2.9%)31.25 ±\pm 8.10 (16.8%)26.00 ±\pm 7.79 (26.0%)27.00 ±\pm 1.41 (14.5%)
Base Refine65.00 ±\pm 5.72 (21.7%)59.00 ±\pm 5.29 (31.7%)58.25 ±\pm 4.72 (58.3%)49.25 ±\pm 3.10 (26.5%)
DafnyPro161.75 ±\pm 5.32 (53.9%)65.75 ±\pm 3.59 (35.3%)62.75 ±\pm 1.89 (62.8%)47.25 ±\pm 2.22 (25.4%)
AxDafny166.50 ±\pm 3.11 (55.5%)122.25 ±\pm 3.69 (65.7%)69.00 ±\pm 0.82 (69.0%)50.75 ±\pm 1.26 (27.3%)
CoVeri181.75 ±\pm 8.42 (60.6%)132.75 ±\pm 7.80 (71.4%)74.25 ±\pm 2.75 (74.3%)58.75 ±\pm 2.87 (31.6%)

These results show positive mean gains on all four benchmarks, while the reported SDs quantify run-to-run variability.

To further examine whether CoVeri remains beneficial with frontier backbones, we added results for GPT-5.6 Luna and Terra across all 300 DafnyComp tasks, including an additional Codex baseline as suggested by reviewer i73Z.

ModelMethodVerifiedSSRTokens/task (k)Requests/task
GPT-5.6 LunaBase83105.281.03
Base Refine2142021.242.94
DafnyPro2271119.602.66
AxDafny2311728.504.27
Codex23712107.549.29
CoVeri2532615.683.85
GPT-5.6 TerraBase112115.161.00
Base Refine2351315.622.37
DafnyPro244414.812.14
AxDafny241624.393.84
Codex2469106.978.52
CoVeri2692815.493.89

CoVeri's advantages persist on both backbones, with the highest Verified and SSR counts among the evaluated methods and much lower recorded token and request costs than the additional Codex baseline.

Second, the ablation gains persist across repeated runs and translate into higher mean verified counts under the reported common token ceilings. The ablation experiments cover both the 35B and 122B backbones, with the original run and three additional runs for each setting.

Method122B35B
VerifiedTokens/task (k)Requests/taskVerifiedTokens/task (k)Requests/task
Full CoVeri181.75 ±\pm 8.4280.99 ±\pm 4.485.43 ±\pm 0.19159.00 ±\pm 6.98110.03 ±\pm 4.966.01 ±\pm 0.18
Single-edit150.50 ±\pm 5.8090.26 ±\pm 1.685.84 ±\pm 0.04124.50 ±\pm 6.45116.66 ±\pm 2.226.36 ±\pm 0.05
w/o TD155.75 ±\pm 2.6381.98 ±\pm 1.735.44 ±\pm 0.08141.00 ±\pm 3.16108.64 ±\pm 0.695.90 ±\pm 0.07
w/o BU153.75 ±\pm 2.75108.75 ±\pm 2.939.89 ±\pm 0.09124.00 ±\pm 10.86146.40 ±\pm 1.7210.60 ±\pm 0.13
CoVeri-init + Base Refine116.50 ±\pm 4.1260.58 ±\pm 0.685.35 ±\pm 0.0498.25 ±\pm 2.6371.75 ±\pm 1.035.58 ±\pm 0.02
Structure-guided149.75 ±\pm 5.5095.60 ±\pm 3.386.05 ±\pm 0.24120.75 ±\pm 3.10133.49 ±\pm 2.127.05 ±\pm 0.09

For both backbones, Full CoVeri outperforms all five controls, while the reported SDs quantify run-to-run variability. To assess CoVeri under common token ceilings, we select the last completed iteration within the dataset-level ceiling for each method and run. This is a retrospective comparison within the recorded five-iteration horizon. A dash indicates that at least one run has no recorded complete iteration within that budget.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summery of rebuttal

Dear Area Chair and Reviewers,

We are grateful to the Area Chair for overseeing the review process and to the reviewers for their constructive comments and questions. We appreciate the reviewers’ questions, which led us to clarify CoVeri’s contribution and extend the evaluation in three ways.

Methodological novelty. All three reviewers asked what CoVeri adds to structure-guided synthesis and verifier-guided repair. We clarified how CoVeri turns unmet caller obligations into explicit targets for jointly synthesizing interface contracts and the internal annotations that prove them. With the 35B backbone, CoVeri outperformed AutoSpec, AutoSpec+, and Preguss on both DafnyComp and SWE-Dafny, as well as ConVer on SWE-Dafny. On DafnyComp, it also outperformed a control that permits joint edits within call-graph groups.

Robustness and stronger backbones. Reviewers asked whether the gains persist across runs, under comparable token budgets, and against a general-purpose coding agent. We ran three additional seeds on DafnyComp with the 122B backbone. Across the original run and the three new runs, CoVeri verified an average of 181.75 of 300 programs, compared with 166.50 for AxDafny, the strongest baseline in this comparison. CoVeri also retained its lead under the common token ceilings evaluated in our four-run ablation analysis. In separate comparisons on DafnyComp, it verified 16 more programs than Codex with GPT-5.6 Luna and 23 more with Terra, using about 85% fewer recorded tokens on both.

Specification quality and benchmark validation. Reviewers asked why only 37 of 175 verified DafnyComp outputs in the original 122B run met the Superior Specifications criterion across all declarations. We audited the other 138 and found that 115 met both reference checks at the entry point, with gaps confined to auxiliary declarations. For SWE-Dafny, all the 100 repaired translations passed all 318 source-repository regression cases, and all 75 eligible pre-repair faulty controls failed verification under the fixed specifications and proof annotations.

Reviewer 4Lay indicated that our response had resolved most of their concerns, which we understand to include their questions about CoVeri’s methodological novelty and our analysis of the Superior Specifications criterion. The revised manuscript presents the method and related-work comparisons in Sections 1–3 and the additional experiments and validation in Section 4 and Appendices B and D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.