acceptodds
Under review as a conference paper at ICLR 2027

“Code is cheap, show me the specification”: Training LLMs for End-to-End Verified Code Generation

Abstract

Formal verification offers strong correctness guarantees for software, but writing precise specifications is often as difficult as proving them. Moreover, weak specifications can satisfy a verifier without capturing meaningful behavior, yielding formally verified yet empty guarantees. Contemporary LLMs lack the ability to generate verified code based on just natural language descriptions. Existing benchmarks majorly focus on evaluating specifications only indirectly, typically through testing or one-sided implication checks over simple pre-/post-conditions. We take a step toward closing both gaps for Rust paired with Verus, a verifier that checks Rust code against developer-written specifications. In this work, we develop techniques for end-to-end training and evaluation of verified code generation in Rust. We introduce SpecBench, a specification-evaluation benchmark with SMT-based equivalence checking that definitively judges generated Verus specifications; develop the first end-to-end training pipeline Verus prover to learn end-to-end verified code generation; and release VerusLM-4B-Codex, a 4B-parameter model trained from Qwen3-4B (Instruct) that significantly outperforms models of comparable size and even some larger ones.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Verified Code Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x5DyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the joint generation of executable code, formal specifications, and proofs in Verus. It introduces SpecBench, a 97-task specification-restoration benchmark, and VerusLM-4B-Codex, trained through multicomponent supervised fine-tuning and multistage verifier-guided reinforcement learning. Experiments cover proof generation, specification restoration, and end-to-end generation across VeruSage-Bench, SpecBench, VTB-XS and VerifyThisBench. The trained model achieves higher verifier pass rates, but semantic validation reveals substantial remaining gaps: it obtains 0% specification-equivalence success on SpecBench and 0% coherence on VerifyThisBench despite a 75.61% verifier pass rate there. These findings highlight persistent reward hacking and the importance of evaluating specification quality alongside proof acceptance.

Critical strengths
  1. Solid experiments. The evaluation covers 4 different benchmarks and compares performance against both general-purpose and specialized models.

  2. Straightforward method. The approach is fairly direct and easy to follow.

Critical weaknesses
  1. Bad writing. The "AI USE STATEMENT" section is missing, which violates the ICLR submission format. Broken references such as "Section ??" and "Table ??" appear in both the main text and the appendix. In Section 4.3, there is a missing line break before "Multi-turn environment." Overall, the paper feels rushed.

  2. After RL, the Plain-Harness metric on VeruSage-Bench and the Coherent metric on VerifyThisBench decline. At the end of Section 5.2.1, the authors also note that "reward-hacking pattern persists whenever the verification signal dominates supervision." This seems to suggest that the task may not be well suited to reinforcement learning for model training, since judging the correctness of model-generated spec is highly difficult.

Questions for authors

Missing necessary ablation or comparison experiments. How do the data mixing ratio and Gated reward affect RL performance? Also, would training methods that are effective on Lean [1] or Dafny [2] outperform the proposed method when applied to Verus?

[1] Goedel-Code-Prover: Hierarchical Proof Search for Open State-of-the-Art Code Verification. arxiv.org/abs/2603.19329

[2] Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny. https://arxiv.org/abs/2507.16331

Reviewer_3q6JOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a Verus benchmark called SpecBench and a post-trained Qwen3-4B Instruct model called VerusLM-4B-Codex trained for verified code generation in Verus. SpecBench draws tasks from existing benchmarks VeruSAGE and VerifyThisBench and re-purposes them by stripping away select portions of the Verus fragments to isolate tasks such as specification generation, verified code synthesis for NL description and others. While VerusLM-4B-Codex is post-trained via SFT and RL on a modified subset of VeruSyn training corpus. The paper evaluates the model-trained amongst other closed and open-weight models on the SpecBench benchmark, as well as others, claiming competitive performance compared to frontier models.

Critical strengths
  1. The paper addresses a critical topic, verifier acceptance is not a meaningful metric on it's own and requires for the specification verified against to be semantically meaningful. The paper introduces a multi-stage evaluation framework anti-cheat constraints to catch specifications that are verified but not semantically equivalent. The evaluation-studies comparing expert reviewers evaluation with Gemini 3 Flash are very welcome and point the field in a good research direction.
Critical weaknesses
  1. The authors claim their trained model outperforms much larger models (“exceeds Kimi-K2.5, a model roughly 250× larger”, based on the Figure 3 average of 33.43% vs. 26.71%). This rests on VerifyThisBench Verified result of 75.61%, which the authors themselves attribute to reward hacking, as the corresponding Coherent score is 0%. Comparing the Coherent metric (which the paper describes as the one that shall be adopted), Kimi 2.5 beats the VerusLM-4B-Codex uniformly across all benchmarks with a wide margin. Aditionally, the paper makes the Stage 2 checker of SpecBench one of its core new contributions to its training pipeline but the low scores on this metric that are even below it's base model (0% to 1.03%). This is a negative result that should be communicated clearly, instead of being described as the model “remaining more robust under stricter checks”.

  2. Novelty and reliability of the benchmark is not convincing. Firstly, the formal Stage 2 equivalence check rarely decides the final outcome from the results. As shown in RQ2, even on clearly equivalent cases it still fails, requiring Stage 3 with a LLM to judge the final outcome. This judge appears very lenient to stronger frontier models (e.g. Kimi-K2.5: 10 cases, total 9.68; GPT-5.5: 27 cases, total 22.41). Second, the closest related work of Verus-SpecBench and Verus-SpecGym is missing is not discussed. Novelty compared to these benchmarks appears limited. Additionally, they report that an LLM judge misses 26% of the specification failures caught by executable tests, which raises even more questions on the reliability of SpecBench’s Stage 3.

Questions for authors
  1. Could the authors explain why does SpecBench Stage 2 drop to 0% after RL with an equivalence reward? How often did the equivalence reward (Phases B/C) actually fire during training?
  2. What is the distribution of Stage 3 judge scores? How often does the judge award partial rather than full credit, and how often does it score a specification below 0.5?
  3. Section 3.3 states that the VeruSAGE repositories are excluded from training “by construction”, but VeruSyn’s agent-trajectory data was collected on VeruSAGE-Bench systems (NRKernel, Storage, Vest), which also appear in SpecBench. Was this data excluded from your training subset, and how was the exclusion enforced?
Reviewer_5BJdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the combination of LLM-based code generation and formal program verification. The core idea is that passing a verifier does not necessarily mean satisfying the actual requirements. To address this issue, the authors propose SpecBench, a benchmark containing specification-generation tasks to evaluate the equivalence and semantic coherence of generated specifications. Moreover, the authors propose a 4B-parameter model, VerusLM-4B-Codex, which incorporates specification-equivalence checking into reinforcement learning to improve verified code generation. The experiments compare the proposed model with various baselines across several benchmarks.

Critical strengths
  • SpecBench addresses an important limitation of verifier-based code generation: a program may pass verification even when its specification does not correctly capture the intended behavior. The proposed SMT-based equivalence harness provides stronger evidence of specification correctness than verifier acceptance or test-based evaluation alone.
  • The combination of multi-task SFT, multi-turn verifier feedback, and RL with structured rewards forms a technically substantial training pipeline. The proposed model demonstrates a large improvement on VerifyThisBench-XS, increasing verification success from 6.06% after SFT to 46.46% after RL. This suggests that specialized training can enable a relatively small model to effectively leverage formal-verification feedback.
Critical weaknesses
  • A central motivation of this work is that verifier acceptance alone cannot ensure meaningful correctness. However, on SpecBench, VerusLM-4B-Codex achieves only 4.12% Stage 1 verification success and 0% Stage 2 formal equivalence. On VerifyThisBench-XS, it obtains 46.46% Verified but only 5.05% Coherent, while on VerifyThisBench, it achieves 75.61% Verified but 0% Coherent. Although failures to prove formal equivalence do not necessarily imply incorrect specifications, the large gap between verification success and semantic coherence raises substantial concerns.
  • The most distinctive aspect of the proposed RL method is the use of formal specification equivalence as an additional reward beyond ordinary verification success. However, the experiments primarily compare the SFT checkpoint with the full RL-trained model, without a controlled comparison between verifier-only RL and verifier-plus-equivalence RL. Consequently, it is unclear whether the reported improvements can be attributed to the proposed equivalence-based reward or to conventional verifier-guided RL.
Questions for authors
  • Why does verification success remain largely disconnected from semantic correctness despite equivalence-aware RL? Can the authors clarify whether equivalence-based rewards provide improvements over verifier-only RL?
  • What is the specific novelty of SpecBench compared with prior specification-evaluation methods such as VERINA and VeriEquivBench, particularly regarding the equivalence-checking methodology and supported specification types?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.